Oct 30 21:43:27.091140 ip-10-0-141-86 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:43:27.091153 ip-10-0-141-86 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:43:27.091163 ip-10-0-141-86 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:43:27.091492 ip-10-0-141-86 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:43:37.163227 ip-10-0-141-86 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:43:37.163244 ip-10-0-141-86 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5641d39c9cc545f2ac9e51c7179c70a0 --
Oct 30 21:45:50.114025 ip-10-0-141-86 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:45:50.525840 ip-10-0-141-86 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:45:50.525840 ip-10-0-141-86 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:45:50.525840 ip-10-0-141-86 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:45:50.525840 ip-10-0-141-86 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:45:50.529729 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529646    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.529729 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529728    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.529729 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529732    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529735    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529738    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529741    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529743    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529746    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529749    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529752    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529754    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529757    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529760    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529763    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529766    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529768    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529771    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529774    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529776    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529786    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529791    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.529821 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529795    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529798    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529800    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529803    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529806    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529808    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529811    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529813    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529816    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529819    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529821    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529824    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529828    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529831    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529834    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529837    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529839    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529842    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529844    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.530251 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529847    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529849    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529852    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529854    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529857    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529859    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529861    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529864    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529866    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529869    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529872    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529877    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529881    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529884    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529888    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529891    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529893    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529896    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529898    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529901    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.530734 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529904    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529906    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529909    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529912    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529914    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529917    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529920    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529923    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529925    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529928    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529930    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529933    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529936    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529938    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529941    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529943    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529946    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529949    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529951    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529954    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.531203 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529956    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529959    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529962    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529964    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529967    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529970    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529973    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529976    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529979    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529981    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529984    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529987    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529989    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529992    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529995    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.529997    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530000    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530003    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530005    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.531707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530008    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530010    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530013    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530016    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530018    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530021    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530024    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530848    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530856    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530859    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530863    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530866    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530869    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530874    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530878    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530881    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530884    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530888    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530891    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530899    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.532374 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530902    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530905    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530908    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530911    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530914    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530916    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530919    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530922    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530924    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530927    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530930    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530932    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530935    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530938    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530940    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530943    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530945    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530948    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530951    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530954    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.533201 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530956    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530959    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530962    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530964    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530967    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530970    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530973    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530975    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530978    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530981    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530983    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530986    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530989    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530992    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530995    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.530997    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531000    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531002    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531005    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.534089 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531007    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531010    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531013    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531015    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531018    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531021    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531023    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531026    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531028    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531031    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531033    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531036    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531039    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531042    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531044    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531047    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531050    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531052    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531055    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531058    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.534921 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531060    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531063    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531066    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531068    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531071    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531074    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531077    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531080    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531083    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531086    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531089    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531092    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531095    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531098    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531100    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531103    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531105    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531108    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531111    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.535624 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531115    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531118    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531121    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531124    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531127    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531130    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531133    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531136    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531139    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531141    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531144    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531146    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531149    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531153    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531156    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531620    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531625    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531628    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531644    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.531696    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.536173 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532212    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532220    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532225    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532229    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532236    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532241    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532245    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532250    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532255    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532260    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532264    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532268    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532273    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532283    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532288    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532292    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532296    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532301    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532306    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532310    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.536840 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532315    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532319    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532323    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532327    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532332    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532336    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532346    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532350    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532355    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532360    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532364    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532369    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532375    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532382    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532388    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532393    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532398    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532402    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532410    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.537344 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532414    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532419    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532423    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532428    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532432    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532437    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532441    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532448    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532453    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532457    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532461    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532471    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532476    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532480    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532484    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532489    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532493    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532497    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532501    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.537897 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532506    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532510    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532515    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532520    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532532    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532537    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532541    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532547    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532552    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532556    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532561    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532566    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532570    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532575    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532579    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532583    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532592    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532597    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532601    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.538440 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532606    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532610    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532614    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532618    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532622    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532626    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532631    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532654    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532659    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532663    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532672    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532676    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532680    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532684    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532688    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532692    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532720    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532752    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532759    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532765    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.538943 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532809    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532918    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532926    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.532930    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533075    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533087    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533098    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533105    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533113    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533118    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533126    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533141    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533147    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533152    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533157    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533162    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533167    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533172    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533177    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533182    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533187    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533192    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533199    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:45:50.539403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533203    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533208    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533212    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533219    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533225    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533230    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533236    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533241    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533247    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533252    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533257    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533262    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533267    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533274    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533279    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533284    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533289    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533294    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533299    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533307    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533312    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533317    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533321    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533327    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:45:50.539956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533334    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533339    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533344    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533349    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533354    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533358    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533364    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533369    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533374    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533378    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533383    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533389    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533399    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533405    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533410    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533415    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533420    2580 flags.go:64] FLAG: --help="false"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533425    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533431    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533436    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533440    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533446    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533452    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:45:50.540626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533457    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533461    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533467    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533471    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533476    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533482    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533487    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533492    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533496    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533502    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533506    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533511    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533515    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533520    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533527    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533537    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533542    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533547    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533552    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533557    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533562    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533570    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533576    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533583    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533588    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:45:50.541257 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533595    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533600    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533605    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533610    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533615    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533620    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533625    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533629    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533659    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533664    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533669    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533674    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533679    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533684    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533688    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533693    2580 flags.go:64] FLAG: --port="10250"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533698    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533703    2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d25aa1dffbad1363"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533709    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533713    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533718    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533723    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533731    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533737    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533742    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533746    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:45:50.541868 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533757    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533762    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533769    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533775    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533780    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533784    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533789    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533794    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533799    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533804    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533809    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533814    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533819    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533824    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533829    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533834    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533839    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533844    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533848    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533853    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533862    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533867    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533872    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533878    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533883    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:45:50.542467 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533887    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533892    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533898    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533903    2580 flags.go:64] FLAG: --v="2"
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533911    2580 flags.go:64] FLAG: --version="false"
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533918    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533924    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.533930    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534147    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534157    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534162    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534166    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534171    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534176    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534180    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534185    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534190    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534194    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534198    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534203    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534207    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.543065 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534212    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534217    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534222    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534226    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534230    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534234    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534238    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534243    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534249    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534256    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534261    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534266    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534270    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534277    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534281    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534286    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534291    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534296    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534300    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.543571 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534304    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534311    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534315    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534319    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534323    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534328    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534332    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534337    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534341    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534345    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534350    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534354    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534358    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534362    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534367    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534372    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534377    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534381    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534385    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534390    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.544060 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534395    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534399    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534403    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534407    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534411    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534416    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534423    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534427    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534431    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534436    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534441    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534445    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534450    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534457    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534461    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534466    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534470    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534474    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534478    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534482    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.544526 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534486    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534491    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534496    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534500    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534504    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534508    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534513    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534517    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534521    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534525    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534531    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534537    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534542    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534547    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534552    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534556    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534561    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534565    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534571    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.545027 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534575    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534580    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534584    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534589    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534596    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534601    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534606    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534611    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534615    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534620    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534624    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534629    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534653    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534658    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534663    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.534707    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:45:50.545481 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534765    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534771    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534777    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534781    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534786    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534790    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534794    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534798    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534803    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534807    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534811    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534816    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534820    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534824    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534828    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534834    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534839    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534843    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534850    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.545882 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534857    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534861    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534867    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534872    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534878    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534882    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534886    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534891    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534895    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534900    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534904    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534908    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534912    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534916    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534920    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534924    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534928    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534932    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534936    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.546350 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534940    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534944    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534948    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534953    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534957    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534962    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534966    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534970    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534975    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534981    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534986    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534990    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534995    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.534999    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535004    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535008    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535013    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535019    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535023    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535028    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.546820 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535032    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535037    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535041    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535045    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535049    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535054    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535058    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535062    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535066    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535071    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535074    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535079    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535083    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535088    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535092    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535096    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535100    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535104    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535108    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535112    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.547301 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535116    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535125    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535131    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535136    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535141    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535145    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535149    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535154    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535158    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535164    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535169    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535173    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535178    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535182    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535186    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535190    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535194    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535198    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535203    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.547782 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535207    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535212    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535216    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535221    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535225    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535230    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535235    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535239    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.535243    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.542148    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.542162    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.542177    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.542181    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542235    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542240    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542244    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542247    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542250    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542253    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542256    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.548223 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542259    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542262    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542265    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542268    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542270    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542273    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542276    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542278    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542281    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542283    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542286    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542289    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542291    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542294    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542298    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542302    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542305    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542308    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542311    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.548750 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542313    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542317    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542322    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542325    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542329    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542332    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542335    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542339    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542342    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542345    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542348    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542350    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542353    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542356    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542359    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542361    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542364    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542367    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542369    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542372    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.549231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542375    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542377    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542380    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542383    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542385    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542388    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542390    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542393    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542395    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542398    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542401    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542403    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542406    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542409    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542411    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542414    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542416    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542419    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542422    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.549726 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542424    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542427    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542429    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542432    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542434    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542437    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542440    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542442    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542445    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542447    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542450    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542453    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542456    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542458    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542461    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542463    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542466    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542468    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542471    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.550158 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542474    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542477    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542480    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542483    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542486    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542489    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542491    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542494    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542497    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542500    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542502    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542505    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542508    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542510    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542513    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542515    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542518    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542521    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542523    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.550600 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542526    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542528    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542531    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.542557    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542723    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542730    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542734    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542737    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542740    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542743    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542745    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542748    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542751    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542754    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542756    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.551074 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542759    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542761    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542764    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542767    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542769    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542772    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542775    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542778    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542780    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542783    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542786    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542788    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542791    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542793    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542796    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542799    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542801    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542804    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542806    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542809    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.551417 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542812    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542814    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542817    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542820    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542823    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542825    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542828    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542830    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542833    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542836    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542839    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542842    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542844    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542847    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542849    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542852    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542854    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542857    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542860    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542862    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.551947 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542865    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542868    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542870    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542873    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542875    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542878    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542881    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542884    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542888    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542891    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542894    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542897    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542900    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542902    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542904    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542907    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542910    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542913    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542916    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.552535 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542918    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542921    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542923    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542926    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542928    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542931    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542933    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542936    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542938    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542941    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542943    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542946    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542949    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542951    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542954    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542956    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542959    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542962    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542964    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.553000 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542968    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542972    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542975    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542978    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542981    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542984    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542986    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542988    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542991    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542993    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542996    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.542999    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543002    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543004    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543007    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543010    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.553441 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543012    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.543036    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543068    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543073    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543076    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543079    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543082    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543084    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543087    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543090    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543093    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543095    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543098    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543101    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543103    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543106    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:45:50.553833 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543108    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543111    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543113    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543116    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543118    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543121    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543123    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543126    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543128    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543131    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543134    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543136    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543140    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543142    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543146    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543148    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543151    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543153    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543156    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543159    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:45:50.554202 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543161    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543164    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543166    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543169    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543172    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543175    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543177    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543180    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543182    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543185    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543188    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543190    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543192    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543195    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543197    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543200    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543203    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543205    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543207    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543210    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:45:50.554670 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543212    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543215    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543218    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543220    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543223    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543232    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543238    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543241    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543244    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543247    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543249    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543252    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543254    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543257    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543260    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543263    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543266    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543268    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543271    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:45:50.555140 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543273    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543276    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543279    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543281    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543284    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543286    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543289    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543292    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543294    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543297    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543299    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543302    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543305    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543307    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543310    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543313    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543315    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543319    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543323    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:45:50.555573 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543328    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543331    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543333    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543336    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543339    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543341    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543344    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543346    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543349    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543352    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543355    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543357    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543361    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:50.543364    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.543414    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.546042    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.546858    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.546959    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:45:50.556109 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.547001    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:45:50.556576 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.549984    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:45:50.556576 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.551200    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:45:50.560892 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.560873    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:45:50.564694 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.564677    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 21:45:50.565421 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.565397    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:45:50.568023 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.568001    2580 fs.go:135] Filesystem UUIDs: map[4384d48a-9e32-4de8-a6d9-72dbbbdf9405:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 abd32526-a01a-4399-b31d-8e59fd29f3a8:/dev/nvme0n1p3]
Oct 30 21:45:50.568093 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.568021    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:45:50.572280 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.572162    2580 manager.go:217] Machine: {Timestamp:2026-10-30 21:45:50.571706334 +0000 UTC m=+0.338651151 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100480 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec295915e20b70bed663a35ef82cfb06 SystemUUID:ec295915-e20b-70be-d663-a35ef82cfb06 BootID:5641d39c-9cc5-45f2-ac9e-51c7179c70a0 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c3:c6:64:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c3:c6:64:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:43:ab:77:02:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:45:50.572852 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.572839    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:45:50.572988 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.572972    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:45:50.577105 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.577085    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:45:50.577247 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.577107    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-86.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:45:50.577291 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.577250    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:45:50.577291 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.577258    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:45:50.577347 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.577294    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:45:50.577347 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.577310    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:45:50.578077 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.578067    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:45:50.581698 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.581686    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:45:50.581816 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.581807    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:45:50.582659 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.582650    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:45:50.582690 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.582663    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:45:50.582690 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.582673    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:45:50.582690 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.582682    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:45:50.582690 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.582689    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:45:50.583047 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.583032    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:45:50.585118 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.585101    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:45:50.589362 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.589349    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:45:50.590106 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590094    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:45:50.590149 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590124    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:45:50.590285 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590274    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:45:50.590313 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590291    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:45:50.590313 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590297    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:45:50.590313 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590302    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:45:50.590313 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590308    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:45:50.590313 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590313    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:45:50.590936 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590927    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:45:50.590971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590936    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:45:50.590971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590944    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:45:50.590971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590950    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:45:50.590971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590958    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:45:50.590971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.590967    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:45:50.593196 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.593187    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:45:50.593196 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.593196    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:45:50.596950 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.596939    2580 server.go:1263] "Started kubelet"
Oct 30 21:45:50.597106 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.597032    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:45:50.597206 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.597123    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:45:50.597486 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.597442    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:45:50.597854 ip-10-0-141-86 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:45:50.599158 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.598832    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:45:50.601431 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.601415    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:45:50.604524 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.604481    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:45:50.604623 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.604524    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:45:50.604847 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.604825    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:50.605322 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605304    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:45:50.605441 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605423    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:45:50.605523 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605313    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:45:50.605581 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605573    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:45:50.605581 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605581    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:45:50.605803 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.605783    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:45:50.605927 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605913    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:45:50.605984 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605930    2580 factory.go:55] Registering systemd factory
Oct 30 21:45:50.605984 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.605948    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:45:50.606531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.606515    2580 factory.go:153] Registering CRI-O factory
Oct 30 21:45:50.606531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.606533    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:45:50.606701 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.606688    2580 factory.go:103] Registering Raw factory
Oct 30 21:45:50.606769 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.606706    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:45:50.607553 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.607534    2580 manager.go:319] Starting recovery of all containers
Oct 30 21:45:50.614925 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.614904    2580 manager.go:324] Recovery completed
Oct 30 21:45:50.620437 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.620421    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:50.622071 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.622039    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:50.622126 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.622079    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:50.622126 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.622093    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:50.622690 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.622671    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:45:50.622775 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.622693    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:45:50.622775 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.622713    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:45:50.622775 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.622676    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:50.623455 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.623438    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:50.623545 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.623470    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:50.623545 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.623480    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:50.625072 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.625055    2580 policy_none.go:50] "Start"
Oct 30 21:45:50.625072 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.625071    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:45:50.625206 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.625082    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:45:50.627194 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.627178    2580 policy_none.go:44] "Start"
Oct 30 21:45:50.658299 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.658264    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:45:50.662367 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.662352    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:45:50.662453 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.662387    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:45:50.662453 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.662399    2580 server.go:87] "Starting device plugin registration server"
Oct 30 21:45:50.662700 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.662686    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:45:50.662776 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.662701    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:45:50.662819 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.662796    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:45:50.662895 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.662881    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:45:50.662895 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.662893    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:45:50.663394 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.663371    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:45:50.663458 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.663426    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:50.741980 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.741937    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:45:50.743604 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.743171    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:45:50.743604 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.743191    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:45:50.743604 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.743210    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:45:50.743604 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.743249    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:45:50.762834 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.762814    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:50.766057 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.766041    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:50.766170 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.766073    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:50.766170 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.766088    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:50.766170 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.766119    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.784427 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.784355    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-86.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:45:50.784545 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.784416    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-86.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:45:50.785704 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.784454    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-86.us-west-2.compute.internal.18e36cd7c4ab2a39  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-86.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-86.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:50.596909625 +0000 UTC m=+0.363854441,LastTimestamp:2026-10-30 21:45:50.596909625 +0000 UTC m=+0.363854441,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-86.us-west-2.compute.internal,}"
Oct 30 21:45:50.792142 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.792121    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-86.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:45:50.792279 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.792251    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:45:50.843345 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.843305    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"]
Oct 30 21:45:50.843474 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.843390    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:50.844398 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.844375    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:50.844511 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.844409    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:50.844511 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.844424    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:50.846488 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.846470    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:50.846613 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.846599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.846711 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.846631    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:50.847285 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.847266    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:50.847368 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.847273    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:50.847368 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.847315    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:50.847368 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.847325    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:50.847368 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.847296    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:50.847492 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.847386    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:50.849316 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.849298    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.849392 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.849328    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:50.850175 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.850161    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:50.850239 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.850185    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:50.850239 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.850199    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:50.871339 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.871316    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:45:50.871447 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.871372    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8zk22"
Oct 30 21:45:50.874922 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.874905    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.878532 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.878517    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.894912 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.894881    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:45:50.895020 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.894955    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-86.us-west-2.compute.internal.18e36cd7c62af968  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-86.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-86.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-86.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:50.622062952 +0000 UTC m=+0.389007769,LastTimestamp:2026-10-30 21:45:50.622062952 +0000 UTC m=+0.389007769,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-86.us-west-2.compute.internal,}"
Oct 30 21:45:50.907402 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.907382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.907488 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.907407    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25a4aaee5f241beed648c981baba69bb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"25a4aaee5f241beed648c981baba69bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.907488 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.907424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:50.957157 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:50.957123    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:45:50.961161 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:50.961142    2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-141-86.us-west-2.compute.internal\" not found" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.008141 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.008110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.008300 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.008151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.008300 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.008175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25a4aaee5f241beed648c981baba69bb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"25a4aaee5f241beed648c981baba69bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.008300 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.008207    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25a4aaee5f241beed648c981baba69bb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"25a4aaee5f241beed648c981baba69bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.008300 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.008216    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.008300 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.008218    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.091898 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.091816    2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-86.us-west-2.compute.internal\" not found" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.162106 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.162070    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:51.163089 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.163060    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:51.163212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.163098    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:51.163212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.163109    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:51.163212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.163140    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.185241 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.185219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.185315 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.185251    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.221596 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.221437    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:51.221596 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.221597    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-86.us-west-2.compute.internal\": node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:51.353574 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.353489    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:51.454189 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.454152    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:51.547769 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.547737    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:45:51.548263 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.547944    2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:45:51.548263 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.547928    2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a26314a26c7f3463b9bcb9cfe0458715-c9cd20b13308b9a1.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-86.us-west-2.compute.internal.18e36cd7c62b7f14\": read tcp 10.0.141.86:44394->3.146.187.184:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-86.us-west-2.compute.internal.18e36cd7c62b7f14  default   5182 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-86.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-86.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-86.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:45:50 +0000 UTC,LastTimestamp:2026-10-30 21:45:50.766093634 +0000 UTC m=+0.533038452,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-86.us-west-2.compute.internal,}"
Oct 30 21:45:51.554415 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.554389    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:51.655184 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.655130    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:51.756195 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.756164    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:51.857711 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:51.857671    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:45:51.878424 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.878360    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:40:51 +0000 UTC" deadline="2028-04-21 12:10:00.2172854 +0000 UTC"
Oct 30 21:45:51.878424 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.878422    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12926h24m8.338867934s"
Oct 30 21:45:51.891314 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.891289    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:45:51.892290 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.892273    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:45:51.892388 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.892304    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:45:51.892388 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.892315    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:45:51.942940 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.942861    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:45:51.951766 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:51.951741    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:45:52.006101 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.006071    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:52.060300 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.060266    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:45:52.070067 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.070039    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:45:52.071475 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.071459    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:45:52.138533 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.138503    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:45:52.397314 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:52.397282    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c08c33d0fce1d0a09d0e1c911723af2.slice/crio-25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309 WatchSource:0}: Error finding container 25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309: Status 404 returned error can't find the container with id 25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309
Oct 30 21:45:52.403952 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.403931    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:45:52.405289 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:52.405269    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a4aaee5f241beed648c981baba69bb.slice/crio-11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287 WatchSource:0}: Error finding container 11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287: Status 404 returned error can't find the container with id 11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287
Oct 30 21:45:52.469783 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.469758    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:45:52.585479 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.585402    2580 apiserver.go:52] "Watching apiserver"
Oct 30 21:45:52.606208 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.606173    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:45:52.700936 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.700908    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:45:52.701218 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.701199    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-gvxg2","kube-system/global-pull-secret-syncer-lwfgr","openshift-multus/multus-additional-cni-plugins-ppjfl","openshift-network-diagnostics/network-check-target-tv4l4","kube-system/konnectivity-agent-blztt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj","openshift-image-registry/node-ca-d66x4","openshift-multus/multus-49fc7","kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-fpgxc","openshift-dns/node-resolver-zccbp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-j6kdw","openshift-network-operator/iptables-alerter-ssjj9"]
Oct 30 21:45:52.705304 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.705272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.708893 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.708871    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:52.708993 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.708958    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:45:52.711252 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.711233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.713520 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.713502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:52.713603 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.713576    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:45:52.715811 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.715792    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:52.718007 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.717980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-host\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.718097 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.718021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-serviceca\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.718097 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.718035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.718097 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.718048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x78p\" (UniqueName: \"kubernetes.io/projected/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-kube-api-access-9x78p\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.718244 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.718111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:52.718244 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.718144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq27q\" (UniqueName: \"kubernetes.io/projected/61248224-57d8-4277-9f87-a0d49e8c4c6f-kube-api-access-qq27q\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:52.720486 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.720472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.723084 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.723059    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.725489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.725462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.725577 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.725532    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:45:52.728013 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.727993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.730589 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.730572    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.733223 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.733204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.748068 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.748024    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"25a4aaee5f241beed648c981baba69bb","Type":"ContainerStarted","Data":"11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287"}
Oct 30 21:45:52.749416 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.749396    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"1c08c33d0fce1d0a09d0e1c911723af2","Type":"ContainerStarted","Data":"25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309"}
Oct 30 21:45:52.770969 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.770947    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bg2hw\""
Oct 30 21:45:52.771301 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.771283    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:45:52.771420 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.771402    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.771491 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.771471    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.771570 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.771557    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:45:52.807466 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.807436    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:45:52.808386 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.808372    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:45:52.808541 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.808530    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rn9hr\""
Oct 30 21:45:52.809163 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.809148    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:45:52.809384 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.809367    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.809619 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.809604    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.809707 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.809692    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:45:52.809707 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.809710    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:45:52.809835 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.809819    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.809874 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.809846    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.810739 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.810246    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqqg2\""
Oct 30 21:45:52.810739 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.810340    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kjn6x\""
Oct 30 21:45:52.813905 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.813879    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:45:52.819349 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:52.819420 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62nzc\" (UniqueName: \"kubernetes.io/projected/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-kube-api-access-62nzc\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.819420 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-log-socket\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.819420 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-cnibin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.819519 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-host\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.819519 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysconfig\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.819519 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.819614 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-k8s-cni-cncf-io\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.819614 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819557    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cnibin\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.819614 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.819557    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:52.819614 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819572    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-host\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.819614 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-device-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.819781 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.819674    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs podName:61248224-57d8-4277-9f87-a0d49e8c4c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:45:53.319653117 +0000 UTC m=+3.086597936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs") pod "network-metrics-daemon-j6kdw" (UID: "61248224-57d8-4277-9f87-a0d49e8c4c6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:52.819781 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-systemd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.819781 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-bin\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.819883 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv67g\" (UniqueName: \"kubernetes.io/projected/67694dfd-17df-4a2a-b598-f03003ee0757-kube-api-access-tv67g\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.819883 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-bin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.819883 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x78p\" (UniqueName: \"kubernetes.io/projected/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-kube-api-access-9x78p\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.819978 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-tmp\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.819978 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-node-log\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.819978 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.819978 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-dbus\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.819978 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-system-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.820123 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.819999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-socket-dir-parent\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.820123 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-kubelet\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.820123 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820045    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.820123 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-serviceca\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.820283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-run\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-var-lib-kubelet\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-agent-certs\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:52.820283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55d27993-063d-499a-b3e8-ef6b2d0ea03e-iptables-alerter-script\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.820400 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-kubelet\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.820400 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-host\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820400 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820362    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.820484 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-netd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.820484 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-etc-kubernetes\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.820484 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.820578 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-systemd\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820578 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:52.820578 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820563    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-serviceca\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.820672 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-conf-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.820672 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.820672 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820639    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtlx\" (UniqueName: \"kubernetes.io/projected/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-kube-api-access-5gtlx\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.820775 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.820775 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-modprobe-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820775 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-sys\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-etc-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.820865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-env-overrides\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.820865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-cni-binary-copy\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.820950 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.820950 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-lib-modules\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820950 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-tuned\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.820950 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d27993-063d-499a-b3e8-ef6b2d0ea03e-host-slash\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.821068 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-system-cni-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.821068 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.820986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4wcq\" (UniqueName: \"kubernetes.io/projected/e5c7e5a7-6225-4513-a2cc-592dfe01d018-kube-api-access-h4wcq\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821068 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-hostroot\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.821068 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-multus-certs\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.821068 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.821212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzp8n\" (UniqueName: \"kubernetes.io/projected/b31df362-1dd4-410c-a946-545f9146c7a6-kube-api-access-bzp8n\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.821212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-netns\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.821212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821098    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.821212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67694dfd-17df-4a2a-b598-f03003ee0757-hosts-file\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.821212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-config\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821212 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-script-lib\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821384 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.821384 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.821384 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovn-node-metrics-cert\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821384 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq27q\" (UniqueName: \"kubernetes.io/projected/61248224-57d8-4277-9f87-a0d49e8c4c6f-kube-api-access-qq27q\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:52.821384 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-conf\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.821531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-konnectivity-ca\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:52.821531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-slash\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-var-lib-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-ovn\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-multus-daemon-config\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.821531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-kubernetes\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.821697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-systemd-units\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821558    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-netns\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.821697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-kubelet-config\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.821697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbplt\" (UniqueName: \"kubernetes.io/projected/e620885a-79e5-483a-9276-e644afb3f27b-kube-api-access-rbplt\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.821865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.821865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67694dfd-17df-4a2a-b598-f03003ee0757-tmp-dir\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.821865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821768    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.821865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p27bh\" (UniqueName: \"kubernetes.io/projected/55d27993-063d-499a-b3e8-ef6b2d0ea03e-kube-api-access-p27bh\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.821865 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.821838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-multus\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.852867 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.852801    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:45:52.852867 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.852860    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xsjfk"
Oct 30 21:45:52.864474 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.864452    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xsjfk"
Oct 30 21:45:52.879560 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.879537    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.879560 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.879553    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.880569 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.880546    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.880697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.880586    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.880697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.880625    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.880826 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.880809    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:45:52.880909 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.880847    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.880909 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.880888    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:45:52.881038 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881024    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:45:52.881094 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881049    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:45:52.881094 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881060    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:45:52.881174 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881156    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:45:52.881228 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881205    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fd5qx\""
Oct 30 21:45:52.881272 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881228    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9dfpb\""
Oct 30 21:45:52.881400 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881388    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mkvf\""
Oct 30 21:45:52.881520 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881506    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5wwct\""
Oct 30 21:45:52.881565 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881556    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pvjtk\""
Oct 30 21:45:52.881606 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881573    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:45:52.881750 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.881738    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:45:52.886787 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.886770    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:45:52.890640 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.890615    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x78p\" (UniqueName: \"kubernetes.io/projected/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-kube-api-access-9x78p\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:52.890750 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.890622    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq27q\" (UniqueName: \"kubernetes.io/projected/61248224-57d8-4277-9f87-a0d49e8c4c6f-kube-api-access-qq27q\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:52.923094 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.923177 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67694dfd-17df-4a2a-b598-f03003ee0757-hosts-file\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.923177 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-config\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.923177 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-script-lib\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.923177 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.923381 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.923381 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923195    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67694dfd-17df-4a2a-b598-f03003ee0757-hosts-file\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.923381 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923247    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.923381 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovn-node-metrics-cert\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.923580 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-conf\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.923580 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-konnectivity-ca\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:52.923580 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-slash\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.923580 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923502    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-slash\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.923580 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-var-lib-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923605    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-conf\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923605    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-ovn\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923659    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-ovn\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-multus-daemon-config\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923748    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-var-lib-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-kubernetes\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-systemd-units\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923812    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-config\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-netns\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923851    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-kubelet-config\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbplt\" (UniqueName: \"kubernetes.io/projected/e620885a-79e5-483a-9276-e644afb3f27b-kube-api-access-rbplt\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923919    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-kubernetes\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67694dfd-17df-4a2a-b598-f03003ee0757-tmp-dir\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.924636 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923999    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27bh\" (UniqueName: \"kubernetes.io/projected/55d27993-063d-499a-b3e8-ef6b2d0ea03e-kube-api-access-p27bh\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924004    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924012    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-script-lib\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-multus\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924005    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-systemd-units\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.923973    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924065    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62nzc\" (UniqueName: \"kubernetes.io/projected/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-kube-api-access-62nzc\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-log-socket\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924137    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-cnibin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysconfig\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924186    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-kubelet-config\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924203    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-konnectivity-ca\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-k8s-cni-cncf-io\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924242    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924254    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.925472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924045    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-netns\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924257    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cnibin\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924317    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-cnibin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924323    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-multus\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924326    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-k8s-cni-cncf-io\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-device-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924354    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysconfig\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924364    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-log-socket\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924367    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67694dfd-17df-4a2a-b598-f03003ee0757-tmp-dir\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924383    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-systemd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924401    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-device-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924401    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cnibin\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-bin\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924438    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-systemd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924454    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924473    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-bin\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv67g\" (UniqueName: \"kubernetes.io/projected/67694dfd-17df-4a2a-b598-f03003ee0757-kube-api-access-tv67g\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924528    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-multus-daemon-config\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926283 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-bin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924673    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-bin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-tmp\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-node-log\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-dbus\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924794    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924794    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-node-log\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-system-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924853    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-dbus\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-socket-dir-parent\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924897    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-system-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924898    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-kubelet\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924935    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-socket-dir-parent\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.924944    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-kubelet\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.925018    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:52.926813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925047    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-run\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.925095    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret podName:14e12922-2bce-41b4-935d-5d3708d1694e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:53.425055427 +0000 UTC m=+3.192000245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret") pod "global-pull-secret-syncer-lwfgr" (UID: "14e12922-2bce-41b4-935d-5d3708d1694e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925121    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-run\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-var-lib-kubelet\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-agent-certs\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925171    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-var-lib-kubelet\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55d27993-063d-499a-b3e8-ef6b2d0ea03e-iptables-alerter-script\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-kubelet\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-host\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925305    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-kubelet\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-netd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925358    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-host\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925397    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925403    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-etc-kubernetes\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925438    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-netd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925480    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-etc-kubernetes\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.927468 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-systemd\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-conf-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925632    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925680    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-systemd\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925825    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55d27993-063d-499a-b3e8-ef6b2d0ea03e-iptables-alerter-script\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925871    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtlx\" (UniqueName: \"kubernetes.io/projected/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-kube-api-access-5gtlx\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925891    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-conf-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.925973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-modprobe-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-sys\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926036    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-etc-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926065    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-modprobe-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928016 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926087    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-env-overrides\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926110    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-etc-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926115    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-sys\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-cni-binary-copy\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926439    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-env-overrides\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926630    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-cni-binary-copy\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926665    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovn-node-metrics-cert\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-lib-modules\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-tuned\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926801    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d27993-063d-499a-b3e8-ef6b2d0ea03e-host-slash\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926825    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-system-cni-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926875    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d27993-063d-499a-b3e8-ef6b2d0ea03e-host-slash\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wcq\" (UniqueName: \"kubernetes.io/projected/e5c7e5a7-6225-4513-a2cc-592dfe01d018-kube-api-access-h4wcq\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-hostroot\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926941    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-multus-certs\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926948    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-hostroot\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.928489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926992    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-multus-certs\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926997    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-tmp\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzp8n\" (UniqueName: \"kubernetes.io/projected/b31df362-1dd4-410c-a946-545f9146c7a6-kube-api-access-bzp8n\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926880    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.927102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-netns\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.926912    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-system-cni-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.927169    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-netns\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.927411    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.927441    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-agent-certs\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.927549    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-lib-modules\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.929026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.928990    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-tuned\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.990972 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.990944    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:45:52.990972 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.990974    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:45:52.991213 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.990990    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lt9qk for pod openshift-network-diagnostics/network-check-target-tv4l4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:52.991213 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:52.991086    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk podName:fee2c691-31c8-4fb6-9e9f-687116de2537 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:53.491064671 +0000 UTC m=+3.258009497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lt9qk" (UniqueName: "kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk") pod "network-check-target-tv4l4" (UID: "fee2c691-31c8-4fb6-9e9f-687116de2537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:52.991923 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.991893    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbplt\" (UniqueName: \"kubernetes.io/projected/e620885a-79e5-483a-9276-e644afb3f27b-kube-api-access-rbplt\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:45:52.992047 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.991902    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27bh\" (UniqueName: \"kubernetes.io/projected/55d27993-063d-499a-b3e8-ef6b2d0ea03e-kube-api-access-p27bh\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:52.992711 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.992681    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62nzc\" (UniqueName: \"kubernetes.io/projected/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-kube-api-access-62nzc\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:52.993064 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.993042    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtlx\" (UniqueName: \"kubernetes.io/projected/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-kube-api-access-5gtlx\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:52.993154 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.993080    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv67g\" (UniqueName: \"kubernetes.io/projected/67694dfd-17df-4a2a-b598-f03003ee0757-kube-api-access-tv67g\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:52.994587 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.994563    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzp8n\" (UniqueName: \"kubernetes.io/projected/b31df362-1dd4-410c-a946-545f9146c7a6-kube-api-access-bzp8n\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:52.995613 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:52.995588    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wcq\" (UniqueName: \"kubernetes.io/projected/e5c7e5a7-6225-4513-a2cc-592dfe01d018-kube-api-access-h4wcq\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:53.018805 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.018773    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:45:53.029122 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.029100    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b567cd8_76eb_4a2e_b05a_b1d8216b7c4f.slice/crio-3151d50d8558f4ceb95b6990e4567bc39ba6d8b3e7dbb249a76be4eb098b98d6 WatchSource:0}: Error finding container 3151d50d8558f4ceb95b6990e4567bc39ba6d8b3e7dbb249a76be4eb098b98d6: Status 404 returned error can't find the container with id 3151d50d8558f4ceb95b6990e4567bc39ba6d8b3e7dbb249a76be4eb098b98d6
Oct 30 21:45:53.036515 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.036500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:45:53.044156 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.044136    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:45:53.045331 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.045307    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4b1c1f_cf24_41bc_a3b9_50ada9e9eb67.slice/crio-f5439ee4318478fa34ff8c2b85c119b0a03d30e3712a1465b4c5516a3864c3b6 WatchSource:0}: Error finding container f5439ee4318478fa34ff8c2b85c119b0a03d30e3712a1465b4c5516a3864c3b6: Status 404 returned error can't find the container with id f5439ee4318478fa34ff8c2b85c119b0a03d30e3712a1465b4c5516a3864c3b6
Oct 30 21:45:53.047308 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.047293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:45:53.049720 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.049697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:45:53.053707 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.053687    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac3b80e_3555_4a21_a9bd_4dec15bcbaed.slice/crio-584a94f52b49f6086fa12e2e932e35d3d05ca98125fbf6419796b31c48e112b4 WatchSource:0}: Error finding container 584a94f52b49f6086fa12e2e932e35d3d05ca98125fbf6419796b31c48e112b4: Status 404 returned error can't find the container with id 584a94f52b49f6086fa12e2e932e35d3d05ca98125fbf6419796b31c48e112b4
Oct 30 21:45:53.058868 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.058845    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d27993_063d_499a_b3e8_ef6b2d0ea03e.slice/crio-62848352f228b49c54a4f4ea85da47a7585e595e04d7df32d381ab0bc5e5290b WatchSource:0}: Error finding container 62848352f228b49c54a4f4ea85da47a7585e595e04d7df32d381ab0bc5e5290b: Status 404 returned error can't find the container with id 62848352f228b49c54a4f4ea85da47a7585e595e04d7df32d381ab0bc5e5290b
Oct 30 21:45:53.059695 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.059674    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31df362_1dd4_410c_a946_545f9146c7a6.slice/crio-709a6375f3498ce1d2d6c1a9d930ca95952d13b3bfab9948d16e7f4717d9c784 WatchSource:0}: Error finding container 709a6375f3498ce1d2d6c1a9d930ca95952d13b3bfab9948d16e7f4717d9c784: Status 404 returned error can't find the container with id 709a6375f3498ce1d2d6c1a9d930ca95952d13b3bfab9948d16e7f4717d9c784
Oct 30 21:45:53.060820 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.060788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:45:53.063473 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.063457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:45:53.070733 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.070705    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99ce2f2_ade1_4d4d_b8d4_a483c4e12f24.slice/crio-b28ae48b14914e57983d77d40e15096780b71f0f53d3c19938bb0eec2ac01dc8 WatchSource:0}: Error finding container b28ae48b14914e57983d77d40e15096780b71f0f53d3c19938bb0eec2ac01dc8: Status 404 returned error can't find the container with id b28ae48b14914e57983d77d40e15096780b71f0f53d3c19938bb0eec2ac01dc8
Oct 30 21:45:53.072075 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.072058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-49fc7"
Oct 30 21:45:53.073980 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.073961    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67694dfd_17df_4a2a_b598_f03003ee0757.slice/crio-3d964f836bc055f3e1d57d522711eb61c5f75fdc7a64d6c1aca0aaf6c1d137d0 WatchSource:0}: Error finding container 3d964f836bc055f3e1d57d522711eb61c5f75fdc7a64d6c1aca0aaf6c1d137d0: Status 404 returned error can't find the container with id 3d964f836bc055f3e1d57d522711eb61c5f75fdc7a64d6c1aca0aaf6c1d137d0
Oct 30 21:45:53.074550 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.074536    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:45:53.081738 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.081704    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode620885a_79e5_483a_9276_e644afb3f27b.slice/crio-d5ad80ae3125aa77f85e1770becaccdd2d70c534715ddc2e2021c705115f48bc WatchSource:0}: Error finding container d5ad80ae3125aa77f85e1770becaccdd2d70c534715ddc2e2021c705115f48bc: Status 404 returned error can't find the container with id d5ad80ae3125aa77f85e1770becaccdd2d70c534715ddc2e2021c705115f48bc
Oct 30 21:45:53.084668 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:45:53.084611    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c7e5a7_6225_4513_a2cc_592dfe01d018.slice/crio-0dcde6fd21a29f326adb968b10cebfff659dd095a09112046b2d824191baea3a WatchSource:0}: Error finding container 0dcde6fd21a29f326adb968b10cebfff659dd095a09112046b2d824191baea3a: Status 404 returned error can't find the container with id 0dcde6fd21a29f326adb968b10cebfff659dd095a09112046b2d824191baea3a
Oct 30 21:45:53.332025 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.331993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:53.332179 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.332108    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:53.332179 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.332160    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs podName:61248224-57d8-4277-9f87-a0d49e8c4c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:45:54.332147404 +0000 UTC m=+4.099092224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs") pod "network-metrics-daemon-j6kdw" (UID: "61248224-57d8-4277-9f87-a0d49e8c4c6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:53.432828 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.432783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:53.433002 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.432972    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:53.433094 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.433051    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret podName:14e12922-2bce-41b4-935d-5d3708d1694e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:54.433031541 +0000 UTC m=+4.199976360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret") pod "global-pull-secret-syncer-lwfgr" (UID: "14e12922-2bce-41b4-935d-5d3708d1694e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:53.533346 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.533314    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:53.533500 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.533471    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:45:53.533500 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.533487    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:45:53.533500 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.533497    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lt9qk for pod openshift-network-diagnostics/network-check-target-tv4l4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:53.533609 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:53.533545    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk podName:fee2c691-31c8-4fb6-9e9f-687116de2537 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:54.533532706 +0000 UTC m=+4.300477510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt9qk" (UniqueName: "kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk") pod "network-check-target-tv4l4" (UID: "fee2c691-31c8-4fb6-9e9f-687116de2537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:53.754030 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.753984    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj" event={"ID":"b31df362-1dd4-410c-a946-545f9146c7a6","Type":"ContainerStarted","Data":"709a6375f3498ce1d2d6c1a9d930ca95952d13b3bfab9948d16e7f4717d9c784"}
Oct 30 21:45:53.756491 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.756462    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ssjj9" event={"ID":"55d27993-063d-499a-b3e8-ef6b2d0ea03e","Type":"ContainerStarted","Data":"62848352f228b49c54a4f4ea85da47a7585e595e04d7df32d381ab0bc5e5290b"}
Oct 30 21:45:53.758345 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.758308    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerStarted","Data":"584a94f52b49f6086fa12e2e932e35d3d05ca98125fbf6419796b31c48e112b4"}
Oct 30 21:45:53.760210 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.760161    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-blztt" event={"ID":"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67","Type":"ContainerStarted","Data":"f5439ee4318478fa34ff8c2b85c119b0a03d30e3712a1465b4c5516a3864c3b6"}
Oct 30 21:45:53.763130 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.763104    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d66x4" event={"ID":"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f","Type":"ContainerStarted","Data":"3151d50d8558f4ceb95b6990e4567bc39ba6d8b3e7dbb249a76be4eb098b98d6"}
Oct 30 21:45:53.765044 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.765023    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"0dcde6fd21a29f326adb968b10cebfff659dd095a09112046b2d824191baea3a"}
Oct 30 21:45:53.766603 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.766577    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49fc7" event={"ID":"e620885a-79e5-483a-9276-e644afb3f27b","Type":"ContainerStarted","Data":"d5ad80ae3125aa77f85e1770becaccdd2d70c534715ddc2e2021c705115f48bc"}
Oct 30 21:45:53.768083 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.768057    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zccbp" event={"ID":"67694dfd-17df-4a2a-b598-f03003ee0757","Type":"ContainerStarted","Data":"3d964f836bc055f3e1d57d522711eb61c5f75fdc7a64d6c1aca0aaf6c1d137d0"}
Oct 30 21:45:53.769633 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.769608    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fpgxc" event={"ID":"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24","Type":"ContainerStarted","Data":"b28ae48b14914e57983d77d40e15096780b71f0f53d3c19938bb0eec2ac01dc8"}
Oct 30 21:45:53.868130 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.868043    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:53 +0000 UTC" deadline="2028-06-30 15:44:55.877800291 +0000 UTC"
Oct 30 21:45:53.868130 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:53.868128    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14609h59m2.009677405s"
Oct 30 21:45:54.341792 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.341755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:54.341992 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.341974    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:54.342060 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.342036    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs podName:61248224-57d8-4277-9f87-a0d49e8c4c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:45:56.342016353 +0000 UTC m=+6.108961179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs") pod "network-metrics-daemon-j6kdw" (UID: "61248224-57d8-4277-9f87-a0d49e8c4c6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:54.443143 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.443104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:54.443337 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.443310    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:54.443414 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.443370    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret podName:14e12922-2bce-41b4-935d-5d3708d1694e nodeName:}" failed. No retries permitted until 2026-10-30 21:45:56.443351239 +0000 UTC m=+6.210296049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret") pod "global-pull-secret-syncer-lwfgr" (UID: "14e12922-2bce-41b4-935d-5d3708d1694e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:54.544596 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.543787    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:54.544596 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.543993    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:45:54.544596 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.544013    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:45:54.544596 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.544028    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lt9qk for pod openshift-network-diagnostics/network-check-target-tv4l4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:54.544596 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.544086    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk podName:fee2c691-31c8-4fb6-9e9f-687116de2537 nodeName:}" failed. No retries permitted until 2026-10-30 21:45:56.544068316 +0000 UTC m=+6.311013128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt9qk" (UniqueName: "kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk") pod "network-check-target-tv4l4" (UID: "fee2c691-31c8-4fb6-9e9f-687116de2537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:54.743872 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.743837    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:54.744073 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.743993    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:45:54.744150 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.744081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:54.744211 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.744164    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:45:54.744270 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.744259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:54.744355 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:54.744334    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:45:54.869128 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.869083    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:53 +0000 UTC" deadline="2028-05-11 20:55:34.082952452 +0000 UTC"
Oct 30 21:45:54.869128 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:54.869126    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13415h9m39.213831438s"
Oct 30 21:45:56.366730 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.364406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:56.366730 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.364627    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:56.366730 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.364719    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs podName:61248224-57d8-4277-9f87-a0d49e8c4c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:46:00.364696673 +0000 UTC m=+10.131641495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs") pod "network-metrics-daemon-j6kdw" (UID: "61248224-57d8-4277-9f87-a0d49e8c4c6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:45:56.469733 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.465978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:56.469733 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.466278    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:56.469733 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.466361    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret podName:14e12922-2bce-41b4-935d-5d3708d1694e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:00.466334894 +0000 UTC m=+10.233279718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret") pod "global-pull-secret-syncer-lwfgr" (UID: "14e12922-2bce-41b4-935d-5d3708d1694e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:45:56.566842 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.566802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:56.567002 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.566983    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:45:56.567087 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.567010    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:45:56.567087 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.567024    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lt9qk for pod openshift-network-diagnostics/network-check-target-tv4l4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:56.567087 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.567085    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk podName:fee2c691-31c8-4fb6-9e9f-687116de2537 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:00.567065138 +0000 UTC m=+10.334009947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt9qk" (UniqueName: "kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk") pod "network-check-target-tv4l4" (UID: "fee2c691-31c8-4fb6-9e9f-687116de2537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:45:56.744965 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.744320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:56.744965 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.744640    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:45:56.744965 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.744758    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:56.744965 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.744834    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:45:56.744965 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.744884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:56.744965 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:56.744941    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:45:56.819372 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.818744    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"25a4aaee5f241beed648c981baba69bb","Type":"ContainerStarted","Data":"12aae623aa5f03893f4c73ece2efea390f20bd454edca06e8ed3785e0556d9dc"}
Oct 30 21:45:56.946569 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:56.946440    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal" podStartSLOduration=4.946419796 podStartE2EDuration="4.946419796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:56.945964262 +0000 UTC m=+6.712909094" watchObservedRunningTime="2026-10-30 21:45:56.946419796 +0000 UTC m=+6.713364626"
Oct 30 21:45:57.829499 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:57.829460    2580 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="694b5606965af93303de405d57b2e8e3e4e1478c9b9d1594febad56a57e574f3" exitCode=0
Oct 30 21:45:57.830023 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:57.829568    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerDied","Data":"694b5606965af93303de405d57b2e8e3e4e1478c9b9d1594febad56a57e574f3"}
Oct 30 21:45:57.835070 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:57.834949    2580 generic.go:356] "Generic (PLEG): container finished" podID="1c08c33d0fce1d0a09d0e1c911723af2" containerID="cd4ac741cdee188b0feac0fb96c093bcdc5da49d4e150bbfcfbbac363971b09c" exitCode=0
Oct 30 21:45:57.836008 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:57.835972    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"1c08c33d0fce1d0a09d0e1c911723af2","Type":"ContainerDied","Data":"cd4ac741cdee188b0feac0fb96c093bcdc5da49d4e150bbfcfbbac363971b09c"}
Oct 30 21:45:58.746960 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:58.746924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:45:58.747162 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:58.747134    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:45:58.747238 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:58.747203    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:45:58.747288 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:58.747263    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:45:58.747342 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:58.747299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:45:58.747391 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:45:58.747360    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:45:58.848409 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:58.848366    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"1c08c33d0fce1d0a09d0e1c911723af2","Type":"ContainerStarted","Data":"366f88a98a53c1ad12dc021ed420299fc2995e49558d18ef9c173db288baf8aa"}
Oct 30 21:45:58.853511 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:58.853426    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-blztt" event={"ID":"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67","Type":"ContainerStarted","Data":"54597fd4df7dadcadddb0c5a202ce82fd8802441a9b902fa1397549e21be1875"}
Oct 30 21:45:59.113142 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:59.113079    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal" podStartSLOduration=7.113057128 podStartE2EDuration="7.113057128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:58.984515261 +0000 UTC m=+8.751460094" watchObservedRunningTime="2026-10-30 21:45:59.113057128 +0000 UTC m=+8.880001959"
Oct 30 21:45:59.866462 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:59.865941    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d66x4" event={"ID":"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f","Type":"ContainerStarted","Data":"9a6aead8d5bf9b034fb55b7e68352dbfba9ec7b4f096d010d21eb09ffdc964a7"}
Oct 30 21:45:59.986715 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:45:59.986660    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-blztt" podStartSLOduration=3.819216864 podStartE2EDuration="8.986626026s" totalImagesPullingTime="5.167409162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:59.113934801 +0000 UTC m=+8.880879646" watchObservedRunningTime="2026-10-30 21:45:59.986626026 +0000 UTC m=+9.753570853"
Oct 30 21:46:00.214451 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.214028    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:46:00.215067 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.214879    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:46:00.346084 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.345582    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d66x4" podStartSLOduration=3.335380559 podStartE2EDuration="9.345564622s" totalImagesPullingTime="6.010184063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:59.987888047 +0000 UTC m=+9.754832874" watchObservedRunningTime="2026-10-30 21:46:00.345564622 +0000 UTC m=+10.112509447"
Oct 30 21:46:00.405913 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.405830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:00.406150 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.406065    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:46:00.406150 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.406127    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs podName:61248224-57d8-4277-9f87-a0d49e8c4c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.406109366 +0000 UTC m=+18.173054183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs") pod "network-metrics-daemon-j6kdw" (UID: "61248224-57d8-4277-9f87-a0d49e8c4c6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:46:00.506797 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.506755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:00.507007 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.506991    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:46:00.507094 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.507057    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret podName:14e12922-2bce-41b4-935d-5d3708d1694e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.507039787 +0000 UTC m=+18.273984604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret") pod "global-pull-secret-syncer-lwfgr" (UID: "14e12922-2bce-41b4-935d-5d3708d1694e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:46:00.607761 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.607472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:00.607761 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.607623    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:46:00.607761 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.607658    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:46:00.607761 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.607671    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lt9qk for pod openshift-network-diagnostics/network-check-target-tv4l4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:46:00.607761 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.607724    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk podName:fee2c691-31c8-4fb6-9e9f-687116de2537 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:08.607707252 +0000 UTC m=+18.374652061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt9qk" (UniqueName: "kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk") pod "network-check-target-tv4l4" (UID: "fee2c691-31c8-4fb6-9e9f-687116de2537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:46:00.745717 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.745009    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:00.745717 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.745116    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:00.745717 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.745188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:00.745717 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.745273    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:00.745717 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.745334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:00.745717 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:00.745410    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:00.870596 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.870560    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj" event={"ID":"b31df362-1dd4-410c-a946-545f9146c7a6","Type":"ContainerStarted","Data":"9b8dde6bec5095618b11bdac32562a547d220f555232223d5943d7e802b7bfef"}
Oct 30 21:46:00.871067 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.870628    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:46:00.871524 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:00.871479    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:46:02.333828 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.333807    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:46:02.679420 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.679314    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:46:02.33382426Z","UUID":"45804373-2f07-4ab5-97ef-bb892ed61847","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:46:02.681108 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.681088    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:46:02.681203 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.681114    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:46:02.745855 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.745828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:02.746010 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:02.745932    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:02.746010 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.745981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:02.746079 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:02.746033    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:02.746112 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.746100    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:02.746192 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:02.746171    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:02.877137 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.877063    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-49fc7" event={"ID":"e620885a-79e5-483a-9276-e644afb3f27b","Type":"ContainerStarted","Data":"89e0999eb58b1d4ab55fd831b416fd5cb39e7f403c90bc9a97bad78ae1260917"}
Oct 30 21:46:02.879946 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.879890    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zccbp" event={"ID":"67694dfd-17df-4a2a-b598-f03003ee0757","Type":"ContainerStarted","Data":"4a775ce53392c52e7454d836f271cbf31aed4b2d48db08cf884b3ab0571497f9"}
Oct 30 21:46:02.882039 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.881967    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj" event={"ID":"b31df362-1dd4-410c-a946-545f9146c7a6","Type":"ContainerStarted","Data":"6606f96622db62a5ea37f486fbe18cd23e58fe3f4421843cfd23e92ea5b51240"}
Oct 30 21:46:02.884015 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.883988    2580 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="68de7a039db3f43bbc062a93d4f1375a398907ffc4685f71c6b3ca1616022c4c" exitCode=0
Oct 30 21:46:02.884106 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.884078    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerDied","Data":"68de7a039db3f43bbc062a93d4f1375a398907ffc4685f71c6b3ca1616022c4c"}
Oct 30 21:46:02.999449 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:02.999249    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-49fc7" podStartSLOduration=3.281470256 podStartE2EDuration="11.999230604s" totalImagesPullingTime="8.717760348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:02.998542411 +0000 UTC m=+12.765487231" watchObservedRunningTime="2026-10-30 21:46:02.999230604 +0000 UTC m=+12.766175433"
Oct 30 21:46:03.116424 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:03.116377    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zccbp" podStartSLOduration=3.002162974 podStartE2EDuration="12.116362748s" totalImagesPullingTime="9.114199774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:03.1162719 +0000 UTC m=+12.883216726" watchObservedRunningTime="2026-10-30 21:46:03.116362748 +0000 UTC m=+12.883307573"
Oct 30 21:46:03.890765 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:03.890730    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ssjj9" event={"ID":"55d27993-063d-499a-b3e8-ef6b2d0ea03e","Type":"ContainerStarted","Data":"2417b72bb6d44090814f07590ebb546a66ff1154aa90d147fcd07a45546d5682"}
Oct 30 21:46:04.029971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.029910    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ssjj9" podStartSLOduration=3.9725492620000002 podStartE2EDuration="13.029889069s" totalImagesPullingTime="9.057339807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:04.029181387 +0000 UTC m=+13.796126220" watchObservedRunningTime="2026-10-30 21:46:04.029889069 +0000 UTC m=+13.796833895"
Oct 30 21:46:04.743822 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.743788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:04.743822 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.743804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:04.744018 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.743931    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:04.744018 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:04.743948    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:04.744092 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:04.744025    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:04.744092 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:04.744060    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:04.893518 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.893483    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fpgxc" event={"ID":"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24","Type":"ContainerStarted","Data":"37d47e570c6ed4b7960125f3912aa678e904b48a183fa7911e78b52f78440d41"}
Oct 30 21:46:04.896114 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.896084    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj" event={"ID":"b31df362-1dd4-410c-a946-545f9146c7a6","Type":"ContainerStarted","Data":"784eba5a609dbbba5db27c8326c04e70f1347fe844d22cf09e81634babc01057"}
Oct 30 21:46:04.898385 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.898356    2580 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="f3b12fc83158184fa6e5d366b6e3817a9b3b740f643a671b0b581ad698d80bf8" exitCode=0
Oct 30 21:46:04.898502 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:04.898441    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerDied","Data":"f3b12fc83158184fa6e5d366b6e3817a9b3b740f643a671b0b581ad698d80bf8"}
Oct 30 21:46:05.013238 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:05.013121    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fpgxc" podStartSLOduration=2.978073303 podStartE2EDuration="14.01310488s" totalImagesPullingTime="11.035031577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:05.012884415 +0000 UTC m=+14.779829240" watchObservedRunningTime="2026-10-30 21:46:05.01310488 +0000 UTC m=+14.780049705"
Oct 30 21:46:05.260224 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:05.260162    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj" podStartSLOduration=3.6999995500000002 podStartE2EDuration="14.260142511s" totalImagesPullingTime="10.560142961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:05.260031072 +0000 UTC m=+15.026975898" watchObservedRunningTime="2026-10-30 21:46:05.260142511 +0000 UTC m=+15.027087348"
Oct 30 21:46:06.744557 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:06.744520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:06.745314 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:06.744591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:06.745314 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:06.744608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:06.745314 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:06.744730    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:06.745314 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:06.744815    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:06.745314 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:06.744849    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:06.904953 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:06.904916    2580 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="85d534889ae0ab84d4e2fbe01645fab86c085a51b8116f8448ff6b9781c16513" exitCode=0
Oct 30 21:46:06.905127 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:06.904971    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerDied","Data":"85d534889ae0ab84d4e2fbe01645fab86c085a51b8116f8448ff6b9781c16513"}
Oct 30 21:46:08.207773 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.207555    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:08.410166 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.410137    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:08.470593 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.470570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:08.470768 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.470750    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:46:08.470831 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.470821    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs podName:61248224-57d8-4277-9f87-a0d49e8c4c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.470799904 +0000 UTC m=+34.237744707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs") pod "network-metrics-daemon-j6kdw" (UID: "61248224-57d8-4277-9f87-a0d49e8c4c6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:46:08.571403 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.571235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:08.571403 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.571401    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:46:08.571552 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.571472    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret podName:14e12922-2bce-41b4-935d-5d3708d1694e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.57145355 +0000 UTC m=+34.338398368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret") pod "global-pull-secret-syncer-lwfgr" (UID: "14e12922-2bce-41b4-935d-5d3708d1694e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:46:08.672234 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.672209    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:08.672411 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.672392    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:46:08.672481 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.672421    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:46:08.672481 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.672435    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lt9qk for pod openshift-network-diagnostics/network-check-target-tv4l4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:46:08.672565 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.672496    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk podName:fee2c691-31c8-4fb6-9e9f-687116de2537 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:24.672476869 +0000 UTC m=+34.439421674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt9qk" (UniqueName: "kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk") pod "network-check-target-tv4l4" (UID: "fee2c691-31c8-4fb6-9e9f-687116de2537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:46:08.743610 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.743583    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:08.743727 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.743610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:08.743727 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.743682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:08.743852 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.743780    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:08.743902 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.743867    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:08.743953 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:08.743938    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:08.911617 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.911592    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:46:08.911953 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.911930    2580 generic.go:356] "Generic (PLEG): container finished" podID="e5c7e5a7-6225-4513-a2cc-592dfe01d018" containerID="2db7728b4e9cf44fcc8885b0c0211f185117d77fe82e5ddf916ad972c35e5978" exitCode=1
Oct 30 21:46:08.912010 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.911982    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"b32b8642408a6d07a4a2198180ab4a37921a9ed89b2bf9f130b64f139972d415"}
Oct 30 21:46:08.912042 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.912036    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"d06d52f28b80ec77ed3f9a17ec56b2fe214fbef079fb3db7c4986a6ceaea5a3d"}
Oct 30 21:46:08.912078 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.912068    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"d7d65b418f93b9babb9b1cb18179208bbedf9daf54715ff348ed104c0c0a1adc"}
Oct 30 21:46:08.912107 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.912098    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"55a7acd4192786b30179a2503f13d38067b00a8d76b1587a592b829965bb83de"}
Oct 30 21:46:08.912143 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.912126    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerDied","Data":"2db7728b4e9cf44fcc8885b0c0211f185117d77fe82e5ddf916ad972c35e5978"}
Oct 30 21:46:08.912170 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:08.912165    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"1c2eb7bb9ee7add6ca7578b7e26bc40d2618ee1e9cd22939a0d9d7f4e3192fd1"}
Oct 30 21:46:09.916549 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:09.916513    2580 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="628d9c9b9ba0e9cf66b9c106985dd17ac92e2a19d841e448479aa94342766386" exitCode=0
Oct 30 21:46:09.917026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:09.916564    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerDied","Data":"628d9c9b9ba0e9cf66b9c106985dd17ac92e2a19d841e448479aa94342766386"}
Oct 30 21:46:10.745431 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:10.745398    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:10.745570 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:10.745500    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:10.745570 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:10.745531    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:10.745570 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:10.745544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:10.745664 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:10.745611    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:10.745733 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:10.745715    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:10.922509 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:10.922471    2580 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="ee3353c39b8ba16b498770738c29f41e02df2c3827aa27480d69e409fc0cb9f8" exitCode=0
Oct 30 21:46:10.923058 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:10.922587    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerDied","Data":"ee3353c39b8ba16b498770738c29f41e02df2c3827aa27480d69e409fc0cb9f8"}
Oct 30 21:46:10.925798 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:10.925778    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:46:10.926124 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:10.926102    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"358789c7658c3ca1a2f3c488eb6d53d87d9184ac2a83cd878872f5176c5266fd"}
Oct 30 21:46:11.931126 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:11.931095    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" event={"ID":"bac3b80e-3555-4a21-a9bd-4dec15bcbaed","Type":"ContainerStarted","Data":"626f48b0300d4b42a5c3200ea201846d0fb57e1b3226ffbc7cc900dc349a6dd7"}
Oct 30 21:46:11.993303 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:11.993275    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:12.058282 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:12.058232    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ppjfl" podStartSLOduration=7.46108987 podStartE2EDuration="21.05821902s" totalImagesPullingTime="13.59712915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:12.058182002 +0000 UTC m=+21.825126830" watchObservedRunningTime="2026-10-30 21:46:12.05821902 +0000 UTC m=+21.825163839"
Oct 30 21:46:12.191387 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:12.191298    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:12.744104 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:12.744073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:12.744248 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:12.744090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:12.744248 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:12.744219    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:12.744348 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:12.744257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:12.744380 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:12.744361    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:12.744456 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:12.744438    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:12.802101 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:12.802070    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:12.995601 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:12.995535    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:13.157782 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:13.157748    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:13.939491 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:13.939466    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:46:13.939821 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:13.939800    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"558cb4c8d08d5735a5e84959e16499635f3c54e336802071678df43d64f6a80e"}
Oct 30 21:46:13.940115 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:13.940097    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:46:13.940248 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:13.940232    2580 scope.go:122] "RemoveContainer" containerID="2db7728b4e9cf44fcc8885b0c0211f185117d77fe82e5ddf916ad972c35e5978"
Oct 30 21:46:13.955934 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:13.955915    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:46:14.743862 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.743830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:14.744444 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.743836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:14.744444 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:14.743929    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:14.744444 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:14.743983    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:14.744444 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.744042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:14.744444 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:14.744151    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:14.945597 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.945572    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:46:14.945947 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.945925    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" event={"ID":"e5c7e5a7-6225-4513-a2cc-592dfe01d018","Type":"ContainerStarted","Data":"367ba38efde12afd2cd71173707a72fffce1a10115e3bf473752caed182282b4"}
Oct 30 21:46:14.946131 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.946114    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:46:14.946186 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.946175    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:46:14.961128 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:14.961102    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:46:15.113859 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.113770    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2" podStartSLOduration=8.982109868 podStartE2EDuration="24.113756464s" totalImagesPullingTime="15.131646596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:15.113557018 +0000 UTC m=+24.880501848" watchObservedRunningTime="2026-10-30 21:46:15.113756464 +0000 UTC m=+24.880701288"
Oct 30 21:46:15.174397 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.174370    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:15.340000 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.339947    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:15.699975 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.699948    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tv4l4"]
Oct 30 21:46:15.700153 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.700057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:15.700222 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:15.700162    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:15.700554 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.700538    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwfgr"]
Oct 30 21:46:15.700628 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.700609    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:15.700710 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:15.700690    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:15.701096 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.701075    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j6kdw"]
Oct 30 21:46:15.701199 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:15.701185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:15.701314 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:15.701296    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:16.377536 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:16.377501    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:16.557384 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:16.557351    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:16.766049 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:16.766019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:16.969716 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:16.969685    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:17.744241 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:17.744209    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:17.744585 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:17.744263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:17.744585 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:17.744274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:17.744585 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:17.744368    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:17.744585 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:17.744448    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:17.744585 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:17.744551    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:17.850838 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:17.850806    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:17.998208 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:17.998100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:19.568309 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:19.568246    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zccbp_67694dfd-17df-4a2a-b598-f03003ee0757/dns-node-resolver/0.log"
Oct 30 21:46:19.704905 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:19.704876    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:19.744528 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:19.744367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:19.744668 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:19.744407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:19.744668 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:19.744608    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:19.744738 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:19.744695    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:19.744738 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:19.744416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:19.744796 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:19.744777    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:20.104770 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:20.104742    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:20.242043 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:20.242018    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d66x4_3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f/node-ca/0.log"
Oct 30 21:46:20.520194 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:20.520164    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:20.688984 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:20.688929    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:20.882233 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:20.882141    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:21.743684 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:21.743650    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:21.744112 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:21.743659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:21.744112 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:21.743777    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:21.744112 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:21.743825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:21.744112 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:21.743907    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:21.744112 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:21.743960    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:21.889289 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:21.889228    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:22.108067 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:22.107967    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:22.282649 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:22.282600    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:22.491289 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:22.491259    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:22.679615 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:22.679582    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:22.841658 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:22.841565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:23.133489 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:23.133366    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:23.371481 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:23.371450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:23.581132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:23.581100    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:23.744336 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:23.744310    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:23.744336 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:23.744337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:23.744542 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:23.744407    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:23.744542 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:23.744421    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:23.744542 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:23.744485    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:23.744628 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:23.744591    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:23.820825 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:23.820797    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:24.079446 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.079414    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:24.233327 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.233271    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:24.383940 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.383834    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:24.488615 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.488583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:24.488799 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.488712    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:46:24.488799 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.488764    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs podName:61248224-57d8-4277-9f87-a0d49e8c4c6f nodeName:}" failed. No retries permitted until 2026-10-30 21:46:56.488749946 +0000 UTC m=+66.255694762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs") pod "network-metrics-daemon-j6kdw" (UID: "61248224-57d8-4277-9f87-a0d49e8c4c6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:46:24.547599 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.547565    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:24.589050 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.589019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:24.589188 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.589130    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:46:24.589188 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.589179    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret podName:14e12922-2bce-41b4-935d-5d3708d1694e nodeName:}" failed. No retries permitted until 2026-10-30 21:46:56.589166909 +0000 UTC m=+66.356111725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret") pod "global-pull-secret-syncer-lwfgr" (UID: "14e12922-2bce-41b4-935d-5d3708d1694e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:46:24.690106 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.690068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:24.690277 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.690235    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:46:24.690277 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.690256    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:46:24.690277 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.690266    2580 projected.go:196] Error preparing data for projected volume kube-api-access-lt9qk for pod openshift-network-diagnostics/network-check-target-tv4l4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:46:24.690395 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:24.690313    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk podName:fee2c691-31c8-4fb6-9e9f-687116de2537 nodeName:}" failed. No retries permitted until 2026-10-30 21:46:56.690301392 +0000 UTC m=+66.457246200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lt9qk" (UniqueName: "kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk") pod "network-check-target-tv4l4" (UID: "fee2c691-31c8-4fb6-9e9f-687116de2537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:46:24.876645 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:24.876614    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:25.418967 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:25.418940    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:25.610794 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:25.610768    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:25.743812 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:25.743741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:25.743812 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:25.743775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:25.744003 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:25.743751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:25.744003 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:25.743852    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:25.744003 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:25.743904    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:25.744003 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:25.743988    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:25.780594 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:25.780567    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:26.078077 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:26.077992    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:26.290023 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:26.289996    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:26.446437 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:26.446401    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:26.603174 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:26.603145    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:26.888030 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:26.887952    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:27.498913 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:27.498886    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:27.744802 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:27.744767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:27.744802 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:27.744789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:27.744994 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:27.744767    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:27.744994 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:27.744866    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:27.744994 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:27.744961    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:27.745088 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:27.745030    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:28.150350 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:28.150309    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:28.325203 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:28.325167    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:28.671696 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:28.671665    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:28.817405 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:28.817373    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:28.979711 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:28.979623    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:29.127060 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:29.127029    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:29.279484 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:29.279395    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:29.743544 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:29.743504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:29.743544 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:29.743537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:29.744085 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:29.743605    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:29.744085 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:29.743671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:29.744085 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:29.743798    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:29.744085 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:29.743860    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:29.902911 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:29.902862    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:30.076585 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:30.076505    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:30.509732 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:30.509684    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:30.667522 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:30.667492    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:30.877459 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:30.877364    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:31.269573 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:31.269542    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:31.467050 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:31.467019    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:31.623179 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:31.623086    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:31.743446 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:31.743397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:31.743619 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:31.743397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:31.743619 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:31.743514    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:31.743619 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:31.743601    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:31.743619 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:31.743397    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:31.743772 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:31.743695    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:33.743971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:33.743938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:33.743971 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:33.743967    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:33.744374 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:33.744061    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:33.744374 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:33.744129    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:33.744374 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:33.744187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:33.744374 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:33.744269    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:35.743608 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:35.743575    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:35.743608 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:35.743605    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:35.744176 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:35.743689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:35.744176 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:35.743694    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:35.744176 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:35.743763    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:35.744176 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:35.743839    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:37.744223 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:37.744169    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:37.744223 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:37.744227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:37.744689 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:37.744170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:37.744689 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:37.744303    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:37.744689 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:37.744382    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:37.744689 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:37.744459    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:39.743680 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:39.743646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:39.744149 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:39.743670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:39.744149 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:39.743846    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6kdw" podUID="61248224-57d8-4277-9f87-a0d49e8c4c6f"
Oct 30 21:46:39.744149 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:39.743676    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:39.744149 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:39.743902    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lwfgr" podUID="14e12922-2bce-41b4-935d-5d3708d1694e"
Oct 30 21:46:39.744149 ip-10-0-141-86 kubenswrapper[2580]: E1030 21:46:39.743966    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tv4l4" podUID="fee2c691-31c8-4fb6-9e9f-687116de2537"
Oct 30 21:46:40.032478 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:40.032397    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:40.200874 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:40.200837    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:40.414186 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:40.414154    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:40.563961 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:40.563927    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:41.069082 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.069054    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeReady"
Oct 30 21:46:41.069478 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.069180    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:46:41.241382 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.241352    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ld584"]
Oct 30 21:46:41.243125 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.243084    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2sdgq"]
Oct 30 21:46:41.243252 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.243182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.244727 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.244708    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xrbvw"]
Oct 30 21:46:41.244830 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.244803    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.246113 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.246095    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ld584"]
Oct 30 21:46:41.246215 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.246135    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2sdgq"]
Oct 30 21:46:41.246272 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.246226    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:41.297409 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.297384    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hs26f\""
Oct 30 21:46:41.297409 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.297384    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:46:41.297596 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.297468    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:46:41.300569 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.300549    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:46:41.301765 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.301743    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrbvw"]
Oct 30 21:46:41.320740 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.320683    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:41.407531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d267e8e-2220-4d18-a54e-9290c579b7bf-config-volume\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.407531 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpn52\" (UniqueName: \"kubernetes.io/projected/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-kube-api-access-wpn52\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:41.407746 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407560    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbbe7220-365b-45da-b66c-9cb581843a2e-crio-socket\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.407746 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl95m\" (UniqueName: \"kubernetes.io/projected/3d267e8e-2220-4d18-a54e-9290c579b7bf-kube-api-access-hl95m\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.407746 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbbe7220-365b-45da-b66c-9cb581843a2e-data-volume\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.407838 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbbe7220-365b-45da-b66c-9cb581843a2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.407838 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtr8\" (UniqueName: \"kubernetes.io/projected/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-api-access-wgtr8\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.407838 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-cert\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:41.407838 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407834    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d267e8e-2220-4d18-a54e-9290c579b7bf-metrics-tls\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.407946 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d267e8e-2220-4d18-a54e-9290c579b7bf-tmp-dir\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.407946 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.407933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.411889 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.411861    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:46:41.411889 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.411879    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-879p5\""
Oct 30 21:46:41.412066 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.411904    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:46:41.412066 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.411983    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:46:41.412066 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.412045    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:46:41.412066 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.412054    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:46:41.412209 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.412076    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:46:41.412209 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.412112    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pz9m9\""
Oct 30 21:46:41.468536 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.468506    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:41.508716 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d267e8e-2220-4d18-a54e-9290c579b7bf-config-volume\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.508716 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpn52\" (UniqueName: \"kubernetes.io/projected/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-kube-api-access-wpn52\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbbe7220-365b-45da-b66c-9cb581843a2e-crio-socket\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl95m\" (UniqueName: \"kubernetes.io/projected/3d267e8e-2220-4d18-a54e-9290c579b7bf-kube-api-access-hl95m\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbbe7220-365b-45da-b66c-9cb581843a2e-data-volume\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508783    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbbe7220-365b-45da-b66c-9cb581843a2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtr8\" (UniqueName: \"kubernetes.io/projected/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-api-access-wgtr8\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-cert\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d267e8e-2220-4d18-a54e-9290c579b7bf-metrics-tls\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d267e8e-2220-4d18-a54e-9290c579b7bf-tmp-dir\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.508935 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.508927    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbbe7220-365b-45da-b66c-9cb581843a2e-crio-socket\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.509397 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.509160    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbbe7220-365b-45da-b66c-9cb581843a2e-data-volume\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.509397 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.509353    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d267e8e-2220-4d18-a54e-9290c579b7bf-tmp-dir\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.509612 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.509592    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.509874 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.509855    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d267e8e-2220-4d18-a54e-9290c579b7bf-config-volume\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.512697 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.512678    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d267e8e-2220-4d18-a54e-9290c579b7bf-metrics-tls\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.512780 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.512706    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbbe7220-365b-45da-b66c-9cb581843a2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.512872 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.512857    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-cert\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:41.572487 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.572432    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl95m\" (UniqueName: \"kubernetes.io/projected/3d267e8e-2220-4d18-a54e-9290c579b7bf-kube-api-access-hl95m\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.572487 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.572471    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtr8\" (UniqueName: \"kubernetes.io/projected/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-api-access-wgtr8\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.572742 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.572724    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpn52\" (UniqueName: \"kubernetes.io/projected/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-kube-api-access-wpn52\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:41.743852 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.743817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:41.744048 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.743873    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:41.744048 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.743994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:41.798994 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.798950    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xlfcr\""
Oct 30 21:46:41.799077 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.799058    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:46:41.799128 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.799107    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:46:41.799128 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.799114    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:46:41.799199 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.799124    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\""
Oct 30 21:46:41.799199 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.799137    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5wdm\""
Oct 30 21:46:41.799199 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.799120    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:46:41.855538 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.855477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:41.862252 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.862237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:46:41.865071 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:41.864937    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:46:42.339685 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:42.339658    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ld584"]
Oct 30 21:46:42.342215 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:42.342197    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xrbvw"]
Oct 30 21:46:42.342771 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:42.342753    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2sdgq"]
Oct 30 21:46:42.385025 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:42.384995    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cb95c2_bb8e_4d26_8e8f_48a49ed0d173.slice/crio-620dcfadd5bd635eebe318ec7c4fe8489c98987722eea8fded7dc638698fc386 WatchSource:0}: Error finding container 620dcfadd5bd635eebe318ec7c4fe8489c98987722eea8fded7dc638698fc386: Status 404 returned error can't find the container with id 620dcfadd5bd635eebe318ec7c4fe8489c98987722eea8fded7dc638698fc386
Oct 30 21:46:42.385469 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:42.385451    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbe7220_365b_45da_b66c_9cb581843a2e.slice/crio-4a8ef7093b01236729bcf699b92f5025a91ee35c7416bcfabe1170f65612fbb9 WatchSource:0}: Error finding container 4a8ef7093b01236729bcf699b92f5025a91ee35c7416bcfabe1170f65612fbb9: Status 404 returned error can't find the container with id 4a8ef7093b01236729bcf699b92f5025a91ee35c7416bcfabe1170f65612fbb9
Oct 30 21:46:42.800022 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:42.799989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:43.003205 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:43.003172    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:43.010991 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:43.010964    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sdgq" event={"ID":"bbbe7220-365b-45da-b66c-9cb581843a2e","Type":"ContainerStarted","Data":"b0fa2e24000153b116a4c71bdd24f539825e2b8badca8e2f959112ef8942ddb1"}
Oct 30 21:46:43.011108 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:43.011018    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sdgq" event={"ID":"bbbe7220-365b-45da-b66c-9cb581843a2e","Type":"ContainerStarted","Data":"4a8ef7093b01236729bcf699b92f5025a91ee35c7416bcfabe1170f65612fbb9"}
Oct 30 21:46:43.012051 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:43.012023    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ld584" event={"ID":"3d267e8e-2220-4d18-a54e-9290c579b7bf","Type":"ContainerStarted","Data":"94c5e9d17c38886cd2d808b1d43e91c1bf6a8a1223923f06297e6b753267a526"}
Oct 30 21:46:43.012960 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:43.012941    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrbvw" event={"ID":"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173","Type":"ContainerStarted","Data":"620dcfadd5bd635eebe318ec7c4fe8489c98987722eea8fded7dc638698fc386"}
Oct 30 21:46:44.017444 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:44.017415    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sdgq" event={"ID":"bbbe7220-365b-45da-b66c-9cb581843a2e","Type":"ContainerStarted","Data":"df4a328881117e4752354da744ff683177b1149ed3f778684260843bbc05c428"}
Oct 30 21:46:44.337072 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:44.336998    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:44.486587 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:44.486509    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:45.022250 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.022210    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ld584" event={"ID":"3d267e8e-2220-4d18-a54e-9290c579b7bf","Type":"ContainerStarted","Data":"86c8d69eb029a999b3171164cb40638ce6e2489d316e2db4cdec7a25fda60834"}
Oct 30 21:46:45.022695 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.022269    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ld584" event={"ID":"3d267e8e-2220-4d18-a54e-9290c579b7bf","Type":"ContainerStarted","Data":"13b524debec8562fb622dde8492d609a53e5cd5acd9912a615908a3fd01f1819"}
Oct 30 21:46:45.022695 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.022353    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:45.123029 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.122998    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:45.143683 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.143621    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-ld584" podStartSLOduration=1.8627907750000001 podStartE2EDuration="4.143608325s" totalImagesPullingTime="2.28081755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:45.142721189 +0000 UTC m=+54.909666015" watchObservedRunningTime="2026-10-30 21:46:45.143608325 +0000 UTC m=+54.910553150"
Oct 30 21:46:45.277116 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.277038    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:45.519993 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.519962    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:45.586074 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.585993    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"]
Oct 30 21:46:45.589311 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.589287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.645558 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.645522    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:46:45.645802 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.645782    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x9s8l\""
Oct 30 21:46:45.645877 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.645828    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:46:45.686992 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.686960    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:45.696517 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.696491    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:46:45.696709 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.696614    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:46:45.696776 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.696754    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:46:45.697690 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.697667    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"]
Oct 30 21:46:45.739857 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.739827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgp7\" (UniqueName: \"kubernetes.io/projected/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-kube-api-access-8vgp7\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.739977 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.739914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.740035 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.739977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.740140 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.740113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.841513 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.841430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.841513 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.841501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgp7\" (UniqueName: \"kubernetes.io/projected/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-kube-api-access-8vgp7\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.841758 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.841585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.841758 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.841615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.842302 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.842270    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.844402 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.844378    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.851121 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.851095    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.878506 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.878477    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f77674798-w874p"]
Oct 30 21:46:45.881831 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.881799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:45.919707 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.919671    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgp7\" (UniqueName: \"kubernetes.io/projected/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-kube-api-access-8vgp7\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:45.937163 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.937130    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:46:45.941908 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.941871    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:46:45.941908 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.941885    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:46:45.942092 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.942008    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:46:45.942191 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.942176    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:46:45.942268 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.942237    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:46:45.942993 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:45.942965    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f77674798-w874p"]
Oct 30 21:46:46.019675 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.019618    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:46:46.026907 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.026840    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xrbvw" event={"ID":"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173","Type":"ContainerStarted","Data":"f5eb3df7086d149c8675f351af296bce129f5622638f94719f86c2f272f8c03e"}
Oct 30 21:46:46.039407 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.039376    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:46:46.039539 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.039441    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cpxgn\""
Oct 30 21:46:46.042726 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.042696    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.042832 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.042751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.042881 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.042841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.042931 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.042908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.042977 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.042933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.043017 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.042983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.043054 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.043018    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.043104 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.043056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.143955 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.143881    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.143955 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.143920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.143955 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.143935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.144208 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.144113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.144208 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.144198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.144307 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.144247    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.144307 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.144281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.144409 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.144306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.144698 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.144671    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.144877 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.144854    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.145499 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.145467    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.145624 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.145604    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.145704 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.145685    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.146604 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.146585    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.147083 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.147058    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.204850 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.204828    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:46:46.208319 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.208295    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.498601 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.498562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:46.573058 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:46.573032    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a3fbfd_209a_4c30_8fb9_9e4812d02f27.slice/crio-40ff8e8394affe59eda38a15264189ca68e7106421637d44b77b26adcae21055 WatchSource:0}: Error finding container 40ff8e8394affe59eda38a15264189ca68e7106421637d44b77b26adcae21055: Status 404 returned error can't find the container with id 40ff8e8394affe59eda38a15264189ca68e7106421637d44b77b26adcae21055
Oct 30 21:46:46.574555 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.574432    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xrbvw" podStartSLOduration=2.247921637 podStartE2EDuration="5.574416069s" totalImagesPullingTime="3.326494432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:46.145169527 +0000 UTC m=+55.912114352" watchObservedRunningTime="2026-10-30 21:46:46.574416069 +0000 UTC m=+56.341360897"
Oct 30 21:46:46.574631 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.574618    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"]
Oct 30 21:46:46.867268 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:46.867241    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f278f63_40f1_40eb_8d8d_04f297071c1f.slice/crio-c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed WatchSource:0}: Error finding container c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed: Status 404 returned error can't find the container with id c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed
Oct 30 21:46:46.868286 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.868268    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f77674798-w874p"]
Oct 30 21:46:46.960408 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:46.960384    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:46:47.029920 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:47.029888    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f77674798-w874p" event={"ID":"9f278f63-40f1-40eb-8d8d-04f297071c1f","Type":"ContainerStarted","Data":"c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed"}
Oct 30 21:46:47.031939 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:47.031915    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2sdgq" event={"ID":"bbbe7220-365b-45da-b66c-9cb581843a2e","Type":"ContainerStarted","Data":"1fbcac5be4e353e2cdd06504fa56cdb2792585a8735a31e8f46e6c3fd9bb41e7"}
Oct 30 21:46:47.032994 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:47.032967    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2" event={"ID":"91a3fbfd-209a-4c30-8fb9-9e4812d02f27","Type":"ContainerStarted","Data":"40ff8e8394affe59eda38a15264189ca68e7106421637d44b77b26adcae21055"}
Oct 30 21:46:47.071400 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:47.071369    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:47.249112 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:47.249079    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:47.302210 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:47.302158    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2sdgq" podStartSLOduration=2.643779574 podStartE2EDuration="6.302144014s" totalImagesPullingTime="3.65836444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:46:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:47.301939208 +0000 UTC m=+57.068884033" watchObservedRunningTime="2026-10-30 21:46:47.302144014 +0000 UTC m=+57.069088839"
Oct 30 21:46:49.042554 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:49.042511    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2" event={"ID":"91a3fbfd-209a-4c30-8fb9-9e4812d02f27","Type":"ContainerStarted","Data":"ecd215debcb9822d999d1a5a887b62554ffd3b7ec7f78045645d0777e2c1cb30"}
Oct 30 21:46:49.043021 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:49.042578    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2" event={"ID":"91a3fbfd-209a-4c30-8fb9-9e4812d02f27","Type":"ContainerStarted","Data":"d84d07aadb199b5ef062a921a54be9d1774e8b8078fba65dcbbaaebac16dd4ce"}
Oct 30 21:46:49.143796 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:49.143760    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:49.165681 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:49.165607    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2" podStartSLOduration=2.259967267 podStartE2EDuration="4.165590484s" totalImagesPullingTime="1.905623217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:49.164688044 +0000 UTC m=+58.931632869" watchObservedRunningTime="2026-10-30 21:46:49.165590484 +0000 UTC m=+58.932535309"
Oct 30 21:46:49.297031 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:49.296946    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:51.050500 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:51.050461    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f77674798-w874p" event={"ID":"9f278f63-40f1-40eb-8d8d-04f297071c1f","Type":"ContainerStarted","Data":"c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc"}
Oct 30 21:46:51.284678 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:51.284649    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:51.453626 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:51.453595    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:51.606166 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:51.606133    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:51.886536 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:51.886461    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:52.030720 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.030686    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:52.287193 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.287162    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:52.287996 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.287961    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f77674798-w874p" podStartSLOduration=3.891113834 podStartE2EDuration="7.287948508s" totalImagesPullingTime="3.396834674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:51.184821312 +0000 UTC m=+60.951766150" watchObservedRunningTime="2026-10-30 21:46:52.287948508 +0000 UTC m=+62.054893386"
Oct 30 21:46:52.288559 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.288545    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"]
Oct 30 21:46:52.292874 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.292859    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.339593 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.339560    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8z59d"]
Oct 30 21:46:52.342570 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.342554    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.351444 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.351417    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"]
Oct 30 21:46:52.394215 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-metrics-client-ca\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394215 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394211    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-root\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394407 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-accelerators-collector-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394407 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.394473 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-sys\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394473 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394450    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqct\" (UniqueName: \"kubernetes.io/projected/c388501b-fe35-4373-ba59-7391b3c79fb5-kube-api-access-ntqct\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394473 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.394554 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394585 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.394619 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cqv\" (UniqueName: \"kubernetes.io/projected/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-api-access-69cqv\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.394682 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.394720 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394685    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.394720 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-textfile\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394779 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-tls\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.394779 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.394741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-wtmp\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.411217 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.411172    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:46:52.411374 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.411273    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9kwx8\""
Oct 30 21:46:52.411374 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.411286    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:46:52.411374 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.411305    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4gblc\""
Oct 30 21:46:52.411374 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.411340    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:46:52.411374 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.411347    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:46:52.416719 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.416696    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:46:52.416926 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.416902    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:46:52.454778 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.454749    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:52.495653 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495604    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-sys\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.495824 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqct\" (UniqueName: \"kubernetes.io/projected/c388501b-fe35-4373-ba59-7391b3c79fb5-kube-api-access-ntqct\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.495824 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.495824 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.495824 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495711    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-sys\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.495824 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.495824 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495767    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cqv\" (UniqueName: \"kubernetes.io/projected/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-api-access-69cqv\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.495824 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495861    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-textfile\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-tls\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-wtmp\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-metrics-client-ca\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-root\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.495978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-accelerators-collector-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496132 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.496007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.496494 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.496164    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-wtmp\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496494 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.496364    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.496494 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.496479    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-textfile\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496654 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.496557    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-root\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.496654 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.496599    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.496867 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.496835    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-metrics-client-ca\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.497075 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.497050    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-accelerators-collector-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.498076 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.498052    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.498542 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.498524    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.498659 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.498627    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.498813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.498792    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-tls\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.498869 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.498853    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.558656 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.558562    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqct\" (UniqueName: \"kubernetes.io/projected/c388501b-fe35-4373-ba59-7391b3c79fb5-kube-api-access-ntqct\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:52.558797 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.558741    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cqv\" (UniqueName: \"kubernetes.io/projected/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-api-access-69cqv\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.608104 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.608073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:46:52.640286 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.640261    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:52.658930 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:52.658900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:46:53.025378 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:53.025346    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87bf1f4_0cf2_4c1d_a513_2c6862b68970.slice/crio-cb2cb86387fd524bfca0708dbd67a51a4538362810bfdfcb1e2fe4668d23e1f0 WatchSource:0}: Error finding container cb2cb86387fd524bfca0708dbd67a51a4538362810bfdfcb1e2fe4668d23e1f0: Status 404 returned error can't find the container with id cb2cb86387fd524bfca0708dbd67a51a4538362810bfdfcb1e2fe4668d23e1f0
Oct 30 21:46:53.025857 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:53.025834    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"]
Oct 30 21:46:53.056962 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:53.056935    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8z59d" event={"ID":"c388501b-fe35-4373-ba59-7391b3c79fb5","Type":"ContainerStarted","Data":"5231c439950f7a74ff3c151c13a8dd9eda810216d4d0ea7a59bc9581a2efd218"}
Oct 30 21:46:53.058094 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:53.058068    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb" event={"ID":"d87bf1f4-0cf2-4c1d-a513-2c6862b68970","Type":"ContainerStarted","Data":"cb2cb86387fd524bfca0708dbd67a51a4538362810bfdfcb1e2fe4668d23e1f0"}
Oct 30 21:46:53.117211 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:53.117171    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:53.491769 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:53.491718    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:53.691826 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:53.691773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:54.401022 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.400989    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:54.443645 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.443549    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:46:54.448270 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.448237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.505388 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.505358    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:46:54.505388 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.505381    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:46:54.505933 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.505381    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:46:54.505933 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.505427    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:46:54.505933 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.505494    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:46:54.505933 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.505498    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:46:54.505933 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.505625    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r29xw\""
Oct 30 21:46:54.510956 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.510928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511105 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.510965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511105 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.510997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511230 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511230 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511230 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511171    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511230 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511190    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511423 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511423 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511423 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrcq\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511423 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511617 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.511617 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.511480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.557726 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.557692    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:46:54.608073 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.608029    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:46:54.612457 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612427    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612605 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612605 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612605 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612605 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612819 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612819 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612667    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612819 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612819 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.612819 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.613047 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.613047 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrcq\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.613047 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.612910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.613190 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.613075    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.613704 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.613648    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.613704 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.613670    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.615755 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.615689    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.615924 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.615899    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.616266 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.616247    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.616357 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.616299    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.616488 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.616469    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.617395 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.617365    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.618035 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.618016    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.640116 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.640087    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:46:54.640116 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.640110    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:46:54.646742 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.646714    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.646863 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.646753    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.675724 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.675693    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrcq\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:54.773909 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:54.773882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:46:55.029813 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.029742    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ld584"
Oct 30 21:46:55.070127 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.070086    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb" event={"ID":"d87bf1f4-0cf2-4c1d-a513-2c6862b68970","Type":"ContainerStarted","Data":"c08dea9463fa0a86a6817726f9464e65ad75e33e4f1f8813a7e219fd446d5caa"}
Oct 30 21:46:55.070301 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.070186    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb" event={"ID":"d87bf1f4-0cf2-4c1d-a513-2c6862b68970","Type":"ContainerStarted","Data":"857ef164f4fba6b43c9d244f4117b29832eb81ef797528d847d36b879ee8cc60"}
Oct 30 21:46:55.070301 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.070211    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb" event={"ID":"d87bf1f4-0cf2-4c1d-a513-2c6862b68970","Type":"ContainerStarted","Data":"f9fb190738a8d1e537a7f3af836e3d3e4fd4d9c0b7a408d921a3ed64b02ffafa"}
Oct 30 21:46:55.071610 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.071579    2580 generic.go:356] "Generic (PLEG): container finished" podID="c388501b-fe35-4373-ba59-7391b3c79fb5" containerID="7bced63265738d8a76cc754c697d75a6bfe2da04fa85068ee2fd061696360be7" exitCode=0
Oct 30 21:46:55.071729 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.071620    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8z59d" event={"ID":"c388501b-fe35-4373-ba59-7391b3c79fb5","Type":"ContainerDied","Data":"7bced63265738d8a76cc754c697d75a6bfe2da04fa85068ee2fd061696360be7"}
Oct 30 21:46:55.198231 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:55.198205    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc41423_0f20_491b_b9ee_fff3583ca471.slice/crio-5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec WatchSource:0}: Error finding container 5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec: Status 404 returned error can't find the container with id 5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec
Oct 30 21:46:55.199415 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.199395    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:46:55.273261 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.273213    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb" podStartSLOduration=1.5355336 podStartE2EDuration="3.273200092s" totalImagesPullingTime="1.737666492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:55.272736436 +0000 UTC m=+65.039681261" watchObservedRunningTime="2026-10-30 21:46:55.273200092 +0000 UTC m=+65.040144916"
Oct 30 21:46:55.598095 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.598054    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:55.746347 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.746317    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:55.908651 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:55.908608    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.075728 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.075693    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerStarted","Data":"5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec"}
Oct 30 21:46:56.077721 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.077688    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8z59d" event={"ID":"c388501b-fe35-4373-ba59-7391b3c79fb5","Type":"ContainerStarted","Data":"59e9537dbbc425d09f9f0bbc7d039b877a0f53489fea2732951bff782f359932"}
Oct 30 21:46:56.077832 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.077735    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8z59d" event={"ID":"c388501b-fe35-4373-ba59-7391b3c79fb5","Type":"ContainerStarted","Data":"090f661681018d0cda9632234d5434049f0703ebe0ff5d478e8a2aa71871ce71"}
Oct 30 21:46:56.199748 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.199628    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8z59d" podStartSLOduration=2.603678834 podStartE2EDuration="4.199614274s" totalImagesPullingTime="1.520457379s" totalInitContainerRuntime="75.478061ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:56.199348692 +0000 UTC m=+65.966293517" watchObservedRunningTime="2026-10-30 21:46:56.199614274 +0000 UTC m=+65.966559143"
Oct 30 21:46:56.361187 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.361147    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.498804 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.498714    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:56.498804 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.498790    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:56.504150 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.504126    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:56.530683 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.530650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:56.537165 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.537139    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.583723 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.583698    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:46:56.593737 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.593709    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:56.633505 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.633452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:56.686525 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.686488    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:46:56.697079 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.697055    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:56.729861 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.729793    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.734886 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.734856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:56.787391 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.787316    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:46:56.787896 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.787874    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"]
Oct 30 21:46:56.791326 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.791304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:56.808566 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.808548    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xlfcr\""
Oct 30 21:46:56.812312 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.812292    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\""
Oct 30 21:46:56.816455 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.816440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:46:56.820607 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.820588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:46:56.835704 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.835669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnxw\" (UniqueName: \"kubernetes.io/projected/8748e87c-9dcc-442b-b739-820276e15f03-kube-api-access-cnnxw\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:56.835814 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.835795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8748e87c-9dcc-442b-b739-820276e15f03-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:56.847293 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.847265    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qzzpd\""
Oct 30 21:46:56.847528 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.847493    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:46:56.848703 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.848681    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:46:56.848802 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.848779    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"]
Oct 30 21:46:56.859326 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.859303    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:56.889299 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.889244    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:56.936751 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.936717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnxw\" (UniqueName: \"kubernetes.io/projected/8748e87c-9dcc-442b-b739-820276e15f03-kube-api-access-cnnxw\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:56.936891 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.936793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8748e87c-9dcc-442b-b739-820276e15f03-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:56.939127 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.939104    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8748e87c-9dcc-442b-b739-820276e15f03-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:56.999066 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:56.999040    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnxw\" (UniqueName: \"kubernetes.io/projected/8748e87c-9dcc-442b-b739-820276e15f03-kube-api-access-cnnxw\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:57.039011 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.038920    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:57.082488 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.082444    2580 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e" exitCode=0
Oct 30 21:46:57.084624 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.083709    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e"}
Oct 30 21:46:57.084624 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.083830    2580 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f77674798-w874p"]
Oct 30 21:46:57.088752 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.088705    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:46:57.104101 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.104076    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:46:57.117875 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.117850    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5wdm\""
Oct 30 21:46:57.126020 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.125997    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:46:57.190904 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.190776    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j6kdw"]
Oct 30 21:46:57.244068 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:57.244033    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e12922_2bce_41b4_935d_5d3708d1694e.slice/crio-5217083d967451f93b8fa6fb8907954480a96f9eccc72a7f1b94d282aee81769 WatchSource:0}: Error finding container 5217083d967451f93b8fa6fb8907954480a96f9eccc72a7f1b94d282aee81769: Status 404 returned error can't find the container with id 5217083d967451f93b8fa6fb8907954480a96f9eccc72a7f1b94d282aee81769
Oct 30 21:46:57.246085 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.245931    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lwfgr"]
Oct 30 21:46:57.475936 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:57.475911    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8748e87c_9dcc_442b_b739_820276e15f03.slice/crio-6b021c8932708178d0aeb8deb35c1c975b3762ad9c5f92d1e4b55cfc0a379038 WatchSource:0}: Error finding container 6b021c8932708178d0aeb8deb35c1c975b3762ad9c5f92d1e4b55cfc0a379038: Status 404 returned error can't find the container with id 6b021c8932708178d0aeb8deb35c1c975b3762ad9c5f92d1e4b55cfc0a379038
Oct 30 21:46:57.477065 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.477042    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"]
Oct 30 21:46:57.497881 ip-10-0-141-86 kubenswrapper[2580]: W1030 21:46:57.497851    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee2c691_31c8_4fb6_9e9f_687116de2537.slice/crio-f4493288d0421badd9f71c2f9cf5cb5390c1fde31ab9d24333711dd6bee04c63 WatchSource:0}: Error finding container f4493288d0421badd9f71c2f9cf5cb5390c1fde31ab9d24333711dd6bee04c63: Status 404 returned error can't find the container with id f4493288d0421badd9f71c2f9cf5cb5390c1fde31ab9d24333711dd6bee04c63
Oct 30 21:46:57.499336 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.499313    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tv4l4"]
Oct 30 21:46:57.805323 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.805234    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:57.992276 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:57.992245    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:46:58.090483 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:58.090360    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h" event={"ID":"8748e87c-9dcc-442b-b739-820276e15f03","Type":"ContainerStarted","Data":"6b021c8932708178d0aeb8deb35c1c975b3762ad9c5f92d1e4b55cfc0a379038"}
Oct 30 21:46:58.092026 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:58.091991    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tv4l4" event={"ID":"fee2c691-31c8-4fb6-9e9f-687116de2537","Type":"ContainerStarted","Data":"f4493288d0421badd9f71c2f9cf5cb5390c1fde31ab9d24333711dd6bee04c63"}
Oct 30 21:46:58.093375 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:58.093317    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lwfgr" event={"ID":"14e12922-2bce-41b4-935d-5d3708d1694e","Type":"ContainerStarted","Data":"5217083d967451f93b8fa6fb8907954480a96f9eccc72a7f1b94d282aee81769"}
Oct 30 21:46:58.094863 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:46:58.094839    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6kdw" event={"ID":"61248224-57d8-4277-9f87-a0d49e8c4c6f","Type":"ContainerStarted","Data":"b8721e15598b631db1c447a28c36a6e53c104e7c059ecfb77767f07676e98c78"}
Oct 30 21:47:00.107247 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.107212    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerStarted","Data":"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"}
Oct 30 21:47:00.107546 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.107337    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerStarted","Data":"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"}
Oct 30 21:47:00.107546 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.107382    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerStarted","Data":"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"}
Oct 30 21:47:00.110372 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.110344    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6kdw" event={"ID":"61248224-57d8-4277-9f87-a0d49e8c4c6f","Type":"ContainerStarted","Data":"7253b1e562b79d9f4d2e27ca8730fb5ddde1ae06283891d62cc906095813c18a"}
Oct 30 21:47:00.110453 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.110399    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6kdw" event={"ID":"61248224-57d8-4277-9f87-a0d49e8c4c6f","Type":"ContainerStarted","Data":"e0d6e9c06f311c492aff007aa1ea4c502b96bdef17a0595dba42fc1db65ed943"}
Oct 30 21:47:00.113084 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.113061    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h" event={"ID":"8748e87c-9dcc-442b-b739-820276e15f03","Type":"ContainerStarted","Data":"bc0add9b8dd057c5158c1fcadf9960811b77974406b02499c0d53e768ea1943e"}
Oct 30 21:47:00.113819 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.113800    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:00.123472 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.123122    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:00.229106 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.229049    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j6kdw" podStartSLOduration=67.436757296 podStartE2EDuration="1m9.229034996s" totalImagesPullingTime="1.7922777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:00.228870644 +0000 UTC m=+69.995815473" watchObservedRunningTime="2026-10-30 21:47:00.229034996 +0000 UTC m=+69.995979820"
Oct 30 21:47:00.346356 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:00.346263    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h" podStartSLOduration=2.0103128 podStartE2EDuration="4.34624845s" totalImagesPullingTime="2.33593565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:00.345749151 +0000 UTC m=+70.112693978" watchObservedRunningTime="2026-10-30 21:47:00.34624845 +0000 UTC m=+70.113193305"
Oct 30 21:47:01.122072 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:01.121757    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerStarted","Data":"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"}
Oct 30 21:47:01.122072 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:01.121850    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerStarted","Data":"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"}
Oct 30 21:47:01.289829 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:01.289773    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:02.129942 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:02.129908    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerStarted","Data":"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"}
Oct 30 21:47:02.131541 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:02.131514    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tv4l4" event={"ID":"fee2c691-31c8-4fb6-9e9f-687116de2537","Type":"ContainerStarted","Data":"125a26b0d3bfc07a75fa14e6c854fba9e9b0d494153d8e34d150a52161dcadf7"}
Oct 30 21:47:02.131770 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:02.131660    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:47:02.159480 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:02.159450    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:02.314999 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:02.314925    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:02.366529 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:02.366468    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.465744402 podStartE2EDuration="8.366444921s" totalImagesPullingTime="5.88824858s" totalInitContainerRuntime="12.451939ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:02.314653425 +0000 UTC m=+72.081598256" watchObservedRunningTime="2026-10-30 21:47:02.366444921 +0000 UTC m=+72.133389751"
Oct 30 21:47:02.434175 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:02.434112    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tv4l4" podStartSLOduration=66.981899214 podStartE2EDuration="1m11.434091242s" totalImagesPullingTime="4.452192028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:02.432413667 +0000 UTC m=+72.199358492" watchObservedRunningTime="2026-10-30 21:47:02.434091242 +0000 UTC m=+72.201036068"
Oct 30 21:47:03.144086 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:03.144047    2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:03.597460 ip-10-0-141-86 kubenswrapper[2580]: I1030 21:47:03.597384    2580 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:47:03.597601 ip-10-0-141-86 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:47:03.610051 ip-10-0-141-86 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:47:03.610228 ip-10-0-141-86 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:47:03.610278 ip-10-0-141-86 systemd[1]: kubelet.service: Consumed 4.100s CPU time.
Oct 30 21:47:03.619260 ip-10-0-141-86 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:47:03.699545 ip-10-0-141-86 kubenswrapper[5503]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:47:03.699545 ip-10-0-141-86 kubenswrapper[5503]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:47:03.699545 ip-10-0-141-86 kubenswrapper[5503]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:47:03.699545 ip-10-0-141-86 kubenswrapper[5503]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:47:03.702402 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702299    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.702402 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702405    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702412    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702418    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702423    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702428    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702433    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702438    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702459    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702464    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702469    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702474    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702478    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702482    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702487    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702491    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702496    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.702496 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702501    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702505    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702510    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702515    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702519    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702523    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702528    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702532    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702536    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702541    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702545    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702549    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702554    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702558    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702563    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702567    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702572    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702576    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702580    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702585    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.702908 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702589    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702593    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702598    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702602    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702614    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702619    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702623    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702628    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702648    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702652    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702656    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702661    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702666    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702670    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702675    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702680    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702684    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702689    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702693    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702699    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.703536 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702704    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702708    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702713    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702717    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702721    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702725    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702729    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702733    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702738    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702742    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702746    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702750    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702754    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702758    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702763    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702773    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702777    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702790    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702795    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702799    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.704109 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702804    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702808    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702812    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702817    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702821    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702825    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702831    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702836    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702843    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702850    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702855    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702860    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702866    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702871    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702879    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702883    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702888    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702892    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702896    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.704581 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702901    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702905    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702909    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702914    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702919    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702923    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702930    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702934    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702941    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.702945    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704711    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704727    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704733    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704739    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704743    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704748    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704752    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704756    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704761    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704766    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.705047 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704770    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704774    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704778    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704783    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704787    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704791    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704794    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704798    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704802    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704807    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704812    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704816    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704820    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704825    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704831    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704835    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704839    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704844    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704848    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.705550 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704853    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704871    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704876    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704881    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704885    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704890    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704895    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704899    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704904    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704909    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704913    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704918    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704922    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704927    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704931    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704941    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704946    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704950    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704955    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.706094 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704959    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704963    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704968    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704974    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704981    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704988    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704993    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.704998    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705004    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705008    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705012    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705016    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705020    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705025    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705030    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705034    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705039    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705043    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705048    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.706555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705054    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705058    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705064    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705068    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705073    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705077    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705082    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705086    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705091    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705095    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705099    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705104    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705108    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705113    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705117    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705122    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705141    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705146    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705150    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705155    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.707030 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705159    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705163    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705168    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705173    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705177    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705182    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705186    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705191    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705196    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705200    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705205    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705210    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705214    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705219    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705228    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705233    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705239    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705244    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.705249    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.707527 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706167    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706178    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706183    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706189    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706195    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706200    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706205    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706209    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706214    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706218    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706223    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706227    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706231    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706236    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706241    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706245    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706249    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706253    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706258    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706263    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.707992 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706267    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706272    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706276    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706281    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706285    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706289    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706301    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706308    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706313    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706318    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706322    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706326    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706331    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706335    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706339    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706343    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706347    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706352    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706356    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.708485 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706361    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706366    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706370    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706375    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706380    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706384    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706390    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706395    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706399    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706404    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706408    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706413    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706417    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706421    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706425    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706429    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706434    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706438    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706443    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706449    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.710398 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706455    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706459    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706463    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706468    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706472    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706477    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706481    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706485    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706489    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706494    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706499    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706504    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706509    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706513    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706517    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706522    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706526    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706531    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706535    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.711165 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706540    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706545    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706549    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706554    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706558    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706563    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706567    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706572    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706576    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706581    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706585    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706589    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706595    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706600    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706605    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706610    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706614    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706619    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706623    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706628    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.711625 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706649    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706654    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706658    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706667    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706673    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706678    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706683    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.706687    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706825    5503 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706837    5503 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706847    5503 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706854    5503 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706861    5503 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706867    5503 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706874    5503 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706881    5503 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706886    5503 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706891    5503 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706897    5503 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706902    5503 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706907    5503 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706912    5503 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:47:03.712170 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706917    5503 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706922    5503 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706930    5503 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706935    5503 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706940    5503 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706946    5503 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706951    5503 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706957    5503 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706963    5503 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706969    5503 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706974    5503 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706979    5503 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706988    5503 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706993    5503 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.706998    5503 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707003    5503 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707008    5503 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707013    5503 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707020    5503 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707025    5503 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707031    5503 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707036    5503 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707041    5503 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707046    5503 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707054    5503 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:47:03.712741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707058    5503 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707063    5503 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707068    5503 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707073    5503 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707080    5503 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707086    5503 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707091    5503 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707096    5503 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707101    5503 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707108    5503 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707113    5503 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707118    5503 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707123    5503 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707129    5503 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707133    5503 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707139    5503 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707144    5503 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707149    5503 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707155    5503 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707161    5503 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707167    5503 flags.go:64] FLAG: --help="false"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707172    5503 flags.go:64] FLAG: --hostname-override="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707178    5503 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707183    5503 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:47:03.713355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707188    5503 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707194    5503 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707200    5503 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707205    5503 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707209    5503 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707215    5503 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707220    5503 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707225    5503 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707230    5503 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707235    5503 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707240    5503 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707245    5503 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707250    5503 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707255    5503 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707260    5503 flags.go:64] FLAG: --lock-file=""
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707264    5503 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707269    5503 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707276    5503 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707285    5503 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707290    5503 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707295    5503 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707300    5503 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707311    5503 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707317    5503 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:47:03.713941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707322    5503 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707327    5503 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707334    5503 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707340    5503 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707346    5503 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707351    5503 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707356    5503 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707361    5503 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707366    5503 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707371    5503 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707376    5503 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707381    5503 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707399    5503 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707405    5503 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707410    5503 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707415    5503 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707419    5503 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707424    5503 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707429    5503 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707434    5503 flags.go:64] FLAG: --port="10250"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707439    5503 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707444    5503 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d25aa1dffbad1363"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707449    5503 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707455    5503 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707460    5503 flags.go:64] FLAG: --register-node="true"
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707467    5503 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:47:03.714509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707473    5503 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707478    5503 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707483    5503 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707488    5503 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707494    5503 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707500    5503 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707506    5503 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707511    5503 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707516    5503 flags.go:64] FLAG: --runonce="false"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707521    5503 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707527    5503 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707532    5503 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707537    5503 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707543    5503 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707548    5503 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707553    5503 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707558    5503 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707562    5503 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707568    5503 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707572    5503 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707577    5503 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707582    5503 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707587    5503 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707593    5503 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707601    5503 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:47:03.715164 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707606    5503 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707611    5503 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707617    5503 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707622    5503 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707628    5503 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707651    5503 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707658    5503 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707663    5503 flags.go:64] FLAG: --v="2"
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707669    5503 flags.go:64] FLAG: --version="false"
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707676    5503 flags.go:64] FLAG: --vmodule=""
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707681    5503 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.707687    5503 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707933    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707943    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707948    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707953    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707958    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707962    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707968    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707972    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707977    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707981    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.716033 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707985    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707990    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707994    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.707999    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708004    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708008    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708012    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708016    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708020    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708024    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708029    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708033    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708038    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708043    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708050    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708057    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708063    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708068    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708073    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.716614 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708077    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708082    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708086    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708091    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708096    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708101    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708105    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708115    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708120    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708124    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708129    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708133    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708138    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708142    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708147    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708151    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708156    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708161    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708165    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.717098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708169    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708174    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708178    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708183    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708187    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708192    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708196    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708201    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708205    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708210    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708214    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708218    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708223    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708227    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708232    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708236    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708240    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708245    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708249    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.717555 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708255    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708261    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708265    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708269    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708274    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708278    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708282    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708287    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708291    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708295    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708299    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708303    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708308    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708312    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708316    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708321    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708326    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708330    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708335    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708339    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.718022 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708344    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708348    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708353    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708358    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708363    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708370    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708375    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708380    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708385    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708389    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708394    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708398    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708403    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708410    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708414    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708419    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708423    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708427    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.718500 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.708432    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.708480    5503 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.709998    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710020    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710027    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710033    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710038    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710043    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710047    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710052    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710056    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710060    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710065    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710069    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710074    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710078    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.718946 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710083    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710088    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710092    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710096    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710100    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710105    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710110    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710115    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710120    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710124    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710129    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710133    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710138    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710142    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710146    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710150    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710155    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710161    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710165    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710170    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.719350 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710174    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710179    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710183    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710188    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710195    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710205    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710211    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710216    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710221    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710226    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710231    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710235    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710240    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710244    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710249    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710253    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710259    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710263    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710268    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.719939 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710273    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710278    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710283    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710288    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710292    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710296    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710301    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710305    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710310    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710314    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710319    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710323    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710327    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710332    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710337    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710341    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710346    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710350    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710354    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.720631 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710359    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710363    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710368    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710372    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710377    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710381    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710385    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710389    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710393    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710398    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710402    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710406    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710411    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710416    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710424    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710429    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710434    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710438    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710443    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710448    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.721193 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710452    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710456    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710460    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710465    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710469    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710475    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710480    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710484    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710489    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710493    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710497    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710502    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710506    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.710510    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.714390    5503 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.714406    5503 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.714423    5503 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.714428    5503 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714484    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714489    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.721727 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714493    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714496    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714498    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714502    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714505    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714508    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714511    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714514    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714517    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714520    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714523    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714525    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714528    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714531    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714533    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714536    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714539    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714542    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714545    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714547    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.722244 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714550    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714553    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714556    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714559    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714561    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714564    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714567    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714570    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714573    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714575    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714578    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714581    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714584    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714587    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714591    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714595    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714598    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714601    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.722912 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714604    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714607    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714609    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714612    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714615    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714617    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714620    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714622    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714625    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714628    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714630    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714655    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714659    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714662    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714665    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714668    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714670    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714673    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714676    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.723554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714679    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714683    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714688    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714691    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714694    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714696    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714699    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714702    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714705    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714708    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714710    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714713    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714717    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714720    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714722    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714725    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714728    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714730    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714732    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.724042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714735    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714738    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714740    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714743    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714745    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714748    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714750    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714754    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714756    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714759    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714761    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714764    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714767    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714769    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714773    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714775    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714778    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714780    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714784    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714786    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.724491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714789    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714791    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714794    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714796    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714799    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714801    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714804    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.714807    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.714835    5503 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715003    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715008    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715011    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715014    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715017    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715020    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715023    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715027    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.725062 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715030    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715033    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715036    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715038    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715041    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715044    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715046    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715049    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715051    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715055    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715057    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715060    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715063    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715066    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715069    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715072    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715074    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715076    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715079    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.725877 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715081    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715085    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715089    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715093    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715096    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715099    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715101    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715104    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715106    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715109    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715112    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715114    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715117    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715119    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715122    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715124    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715127    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715129    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715132    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.726608 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715135    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715137    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715141    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715145    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715148    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715151    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715154    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715156    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715159    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715162    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715164    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715167    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715170    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715172    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715175    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715178    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715181    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715184    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715187    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.727213 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715189    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715192    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715194    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715197    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715200    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715202    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715205    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715207    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715210    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715212    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715215    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715218    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715220    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715223    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715225    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715228    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715231    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715233    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715236    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.727688 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715239    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715242    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715244    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715247    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715250    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715252    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715255    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715258    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715260    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715263    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715265    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715268    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715271    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715273    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715276    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715278    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715281    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715283    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715286    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.728180 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715288    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715291    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715293    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.715316    5503 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715347    5503 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715352    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715356    5503 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715360    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715363    5503 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715365    5503 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715368    5503 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715373    5503 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715377    5503 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715380    5503 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715385    5503 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:47:03.728754 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715391    5503 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715396    5503 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715399    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715402    5503 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715405    5503 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715408    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715410    5503 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715413    5503 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715416    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715418    5503 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715421    5503 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715424    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715429    5503 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715433    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715437    5503 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715441    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715445    5503 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715450    5503 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715454    5503 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715458    5503 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:47:03.729136 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715462    5503 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715466    5503 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715470    5503 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715475    5503 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715479    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715484    5503 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715488    5503 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715493    5503 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715498    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715502    5503 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715506    5503 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715510    5503 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715517    5503 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715522    5503 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715528    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715532    5503 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715537    5503 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715542    5503 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715546    5503 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:47:03.729617 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715550    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715554    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715559    5503 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715564    5503 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715570    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715575    5503 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715580    5503 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715585    5503 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715589    5503 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715594    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715598    5503 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715602    5503 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715607    5503 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715612    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715617    5503 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715621    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715626    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715631    5503 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715651    5503 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715656    5503 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:47:03.730098 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715660    5503 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715664    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715668    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715673    5503 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715678    5503 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715685    5503 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715690    5503 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715695    5503 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715699    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715703    5503 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715708    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715713    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715717    5503 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715721    5503 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715726    5503 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715730    5503 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715734    5503 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715738    5503 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715746    5503 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:47:03.730602 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715754    5503 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715759    5503 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715764    5503 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715769    5503 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715775    5503 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715780    5503 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715786    5503 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715790    5503 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715796    5503 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715801    5503 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715806    5503 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715811    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715819    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715825    5503 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715830    5503 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715834    5503 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:03.715839    5503 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.715906    5503 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.717081    5503 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.717135    5503 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:47:03.731110 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.717327    5503 server.go:1014] "Starting client certificate rotation"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.717466    5503 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.717528    5503 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:40:51 +0000 UTC" deadline="2028-07-25 08:44:17.19815808 +0000 UTC"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.717562    5503 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15202h57m13.480598631s"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.717893    5503 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.719006    5503 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.720347    5503 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.722365    5503 log.go:25] "Validated CRI v1 image API"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.723171    5503 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:47:03.731591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.725511    5503 fs.go:135] Filesystem UUIDs: map[4384d48a-9e32-4de8-a6d9-72dbbbdf9405:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 abd32526-a01a-4399-b31d-8e59fd29f3a8:/dev/nvme0n1p3]
Oct 30 21:47:03.731901 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.725780    5503 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0dcde6fd21a29f326adb968b10cebfff659dd095a09112046b2d824191baea3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0dcde6fd21a29f326adb968b10cebfff659dd095a09112046b2d824191baea3a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3151d50d8558f4ceb95b6990e4567bc39ba6d8b3e7dbb249a76be4eb098b98d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3151d50d8558f4ceb95b6990e4567bc39ba6d8b3e7dbb249a76be4eb098b98d6/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d964f836bc055f3e1d57d522711eb61c5f75fdc7a64d6c1aca0aaf6c1d137d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d964f836bc055f3e1d57d522711eb61c5f75fdc7a64d6c1aca0aaf6c1d137d0/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40ff8e8394affe59eda38a15264189ca68e7106421637d44b77b26adcae21055/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40ff8e8394affe59eda38a15264189ca68e7106421637d44b77b26adcae21055/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a8ef7093b01236729bcf699b92f5025a91ee35c7416bcfabe1170f65612fbb9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a8ef7093b01236729bcf699b92f5025a91ee35c7416bcfabe1170f65612fbb9/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5217083d967451f93b8fa6fb8907954480a96f9eccc72a7f1b94d282aee81769/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5217083d967451f93b8fa6fb8907954480a96f9eccc72a7f1b94d282aee81769/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5231c439950f7a74ff3c151c13a8dd9eda810216d4d0ea7a59bc9581a2efd218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5231c439950f7a74ff3c151c13a8dd9eda810216d4d0ea7a59bc9581a2efd218/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/584a94f52b49f6086fa12e2e932e35d3d05ca98125fbf6419796b31c48e112b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/584a94f52b49f6086fa12e2e932e35d3d05ca98125fbf6419796b31c48e112b4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/620dcfadd5bd635eebe318ec7c4fe8489c98987722eea8fded7dc638698fc386/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/620dcfadd5bd635eebe318ec7c4fe8489c98987722eea8fded7dc638698fc386/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62848352f228b49c54a4f4ea85da47a7585e595e04d7df32d381ab0bc5e5290b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62848352f228b49c54a4f4ea85da47a7585e595e04d7df32d381ab0bc5e5290b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b021c8932708178d0aeb8deb35c1c975b3762ad9c5f92d1e4b55cfc0a379038/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b021c8932708178d0aeb8deb35c1c975b3762ad9c5f92d1e4b55cfc0a379038/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/709a6375f3498ce1d2d6c1a9d930ca95952d13b3bfab9948d16e7f4717d9c784/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/709a6375f3498ce1d2d6c1a9d930ca95952d13b3bfab9948d16e7f4717d9c784/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94c5e9d17c38886cd2d808b1d43e91c1bf6a8a1223923f06297e6b753267a526/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94c5e9d17c38886cd2d808b1d43e91c1bf6a8a1223923f06297e6b753267a526/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8721e15598b631db1c447a28c36a6e53c104e7c059ecfb77767f07676e98c78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8721e15598b631db1c447a28c36a6e53c104e7c059ecfb77767f07676e98c78/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb2cb86387fd524bfca0708dbd67a51a4538362810bfdfcb1e2fe4668d23e1f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb2cb86387fd524bfca0708dbd67a51a4538362810bfdfcb1e2fe4668d23e1f0/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5ad80ae3125aa77f85e1770becaccdd2d70c534715ddc2e2021c705115f48bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5ad80ae3125aa77f85e1770becaccdd2d70c534715ddc2e2021c705115f48bc/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4493288d0421badd9f71c2f9cf5cb5390c1fde31ab9d24333711dd6bee04c63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4493288d0421badd9f71c2f9cf5cb5390c1fde31ab9d24333711dd6bee04c63/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5439ee4318478fa34ff8c2b85c119b0a03d30e3712a1465b4c5516a3864c3b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5439ee4318478fa34ff8c2b85c119b0a03d30e3712a1465b4c5516a3864c3b6/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14e12922-2bce-41b4-935d-5d3708d1694e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/14e12922-2bce-41b4-935d-5d3708d1694e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f/volumes/kubernetes.io~projected/kube-api-access-9x78p:{mountpoint:/var/lib/kubelet/pods/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f/volumes/kubernetes.io~projected/kube-api-access-9x78p major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d267e8e-2220-4d18-a54e-9290c579b7bf/volumes/kubernetes.io~projected/kube-api-access-hl95m:{mountpoint:/var/lib/kubelet/pods/3d267e8e-2220-4d18-a54e-9290c579b7bf/volumes/kubernetes.io~projected/kube-api-access-hl95m major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d267e8e-2220-4d18-a54e-9290c579b7bf/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3d267e8e-2220-4d18-a54e-9290c579b7bf/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55d27993-063d-499a-b3e8-ef6b2d0ea03e/volumes/kubernetes.io~projected/kube-api-access-p27bh:{mountpoint:/var/lib/kubelet/pods/55d27993-063d-499a-b3e8-ef6b2d0ea03e/volumes/kubernetes.io~projected/kube-api-access-p27bh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61248224-57d8-4277-9f87-a0d49e8c4c6f/volumes/kubernetes.io~projected/kube-api-access-qq27q:{mountpoint:/var/lib/kubelet/pods/61248224-57d8-4277-9f87-a0d49e8c4c6f/volumes/kubernetes.io~projected/kube-api-access-qq27q major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61248224-57d8-4277-9f87-a0d49e8c4c6f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/61248224-57d8-4277-9f87-a0d49e8c4c6f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67694dfd-17df-4a2a-b598-f03003ee0757/volumes/kubernetes.io~projected/kube-api-access-tv67g:{mountpoint:/var/lib/kubelet/pods/67694dfd-17df-4a2a-b598-f03003ee0757/volumes/kubernetes.io~projected/kube-api-access-tv67g major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8748e87c-9dcc-442b-b739-820276e15f03/volumes/kubernetes.io~projected/kube-api-access-cnnxw:{mountpoint:/var/lib/kubelet/pods/8748e87c-9dcc-442b-b739-820276e15f03/volumes/kubernetes.io~projected/kube-api-access-cnnxw major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8748e87c-9dcc-442b-b739-820276e15f03/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/8748e87c-9dcc-442b-b739-820276e15f03/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~projected/kube-api-access-8vgp7:{mountpoint:/var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~projected/kube-api-access-8vgp7 major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~projected/kube-api-access-4m925:{mountpoint:/var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~projected/kube-api-access-4m925 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/alertmanager/9 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/config-reloader/6 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~empty-dir/config-out major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~projected/kube-api-access-jbrcq:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~projected/kube-api-access-jbrcq major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~projected/tls-assets major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/config-volume major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/web-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b31df362-1dd4-410c-a946-545f9146c7a6/volumes/kubernetes.io~projected/kube-api-access-bzp8n:{mountpoint:/var/lib/kubelet/pods/b31df362-1dd4-410c-a946-545f9146c7a6/volumes/kubernetes.io~projected/kube-api-access-bzp8n major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bac3b80e-3555-4a21-a9bd-4dec15bcbaed/volumes/kubernetes.io~projected/kube-api-access-5gtlx:{mountpoint:/var/lib/kubelet/pods/bac3b80e-3555-4a21-a9bd-4dec15bcbaed/volumes/kubernetes.io~projected/kube-api-access-5gtlx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbbe7220-365b-45da-b66c-9cb581843a2e/volumes/kubernetes.io~projected/kube-api-access-wgtr8:{mountpoint:/var/lib/kubelet/pods/bbbe7220-365b-45da-b66c-9cb581843a2e/volumes/kubernetes.io~projected/kube-api-access-wgtr8 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbbe7220-365b-45da-b66c-9cb581843a2e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/bbbe7220-365b-45da-b66c-9cb581843a2e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~projected/kube-api-access-ntqct:{mountpoint:/var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~projected/kube-api-access-ntqct major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~projected/kube-api-access-69cqv:{mountpoint:/var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~projected/kube-api-access-69cqv major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173/volumes/kubernetes.io~projected/kube-api-access-wpn52:{mountpoint:/var/lib/kubelet/pods/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173/volumes/kubernetes.io~projected/kube-api-access-wpn52 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volumes/kubernetes.io~projected/kube-api-access-h4wcq:{mountpoint:/var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volumes/kubernetes.io~projected/kube-api-access-h4wcq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e620885a-79e5-483a-9276-e644afb3f27b/volumes/kubernetes.io~projected/kube-api-access-rbplt:{mountpoint:/var/lib/kubelet/pods/e620885a-79e5-483a-9276-e644afb3f27b/volumes/kubernetes.io~projected/kube-api-access-rbplt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~projected/kube-api-access-62nzc:{mountpoint:/var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~projected/kube-api-access-62nzc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fee2c691-31c8-4fb6-9e9f-687116de2537/volumes/kubernetes.io~projected/kube-api-access-lt9qk:{mountpoint:/var/lib/kubelet/pods/fee2c691-31c8-4fb6-9e9f-687116de2537/volumes/kubernetes.io~projected/kube-api-access-lt9qk major:0 minor:340 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/74901337da0882892dc1d799c728e77af7129fcd58ac6e84ac06c8928ca9b522/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/56ac6b35c185bf244c9bf15cb02daec54583c31165973c4428e62f2f8d1a33f3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/12c9dfc97dbafe1dd466dcc66b3ce71f7e6c0ea7a1189643154c30e942da4585/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9d9eb42d06b10a0129b21a9432093900f04f540d4fe573c156b19d3ce6da23f8/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3a2433679096459e63e7c83d2e3003d0a8a95e28ee0b585148db135f8731bc57/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/24d89afe288ec7d3d2bfa610c321f74f31dd8d20883f1f9f069b6f1922fb87be/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/473f40c6a11a8e3bc6e3c0b09721eb01bc2b16314aad9192ef91c8f0456b7d4f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/84690d00ff1caef03d4cc96f4bddaa4f5a29010104ccba51116412a113b6c254/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5ce83e69063be3439431803fb955cdae681ff552a601269ccb228a10317eb4ac/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/914a7700ace922dfdf8bd2150cbd92e9530cbe36847115a60c38ddf4e136b3f4/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e4c8b8ff5d5d16578a4e751e00dc0613a18b457aace77aafc69740e5d2099e09/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/f6bc037c8b7ff2642373cac94e9c2ff710a8cc294ede079a6b482d9d14a5e638/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fd01e9c1ebcd4d8ea2728ef39780c635973a30fe9a007d6ffa355dbeba72e7eb/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5050afe19b4d851bd6099a5f9a08c6ba4a090d362c6cbeb6b6ad832b7d438a30/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bfae8800b7b56d81888372b6f18519c149ba409ee726f72982d75143ebbed608/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0127dbad587e3df554b0dcf7f943bf973f0a5dbf0bbe915267074c1d50144aa2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9585575b37cbc215a03c060a07252cbb380d1dcd85c9cd7d6d08a4038704d743/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7ff2e19d7d23d4171cf7943b55d2c57678dd209b5f8e57f1e3978fe795dd79b1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/febead0a2aef12c170cf3a3fd1645d6f795d6a0def0696bebd0bc9bbee4a6dcc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/197a12e348317a0de3217340105d9e36f1362a26d1adf7651943d646b101fd2b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/14b91d892616655abe8473847812710808383c4b60a676e9fc91caa23646716e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4c21c017a535fc096d10d09477e95228b7b260b266b1a659a78ab56e8ec47e89/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/389adf18de80d04fdd8c234e28fcb7f7f42066b9bd841c8cd64386482d72be82/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f8e98031a42fb9e9b3acfbef7b8670b4e9ab12f1dcd7dcc5992e80aafcf11c66/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ccef82aa6a58541c8eac01627fd72987c050a29295efce2b568d8284163628a8/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/ee5bf0e9930ab6edd4ca05d4c096dcaa06d1522efe71b5af023d354c33b083c9/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/2c0307bfeb53068a75ac73b70d28ce5b6f2cae5687412e49daf4f6d3ee440800/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/7e02ecc7df0dd91bc3aeeee8cf598ed893d57419f61a34e649aa2fca8fffacc0/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/0906555376febd254d426cacf154aaf72dbd3c2b300eeedf8f5a11b748ddbdd5/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/0c0805a346dbec51b32d3e2cd33da8dc3e146027452db02c6e45741e7098b4dd/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/198db12cd618cf737b3742000439208691fc5de52a9bf4a835d75f0b876f2e8f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/b3c2a071a8162f55e28781ce5b91dc19b3942963a3baa5e9abb3339cf51445de/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/8b6ded343428be4b8c259f8cc273dda6985c9bb37c426b5a6152876cdea1cd5a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/8724a94d4897102ac7060dbefa89bbad5677e6c55754c55c292728ff5149ff34/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/e5260c8dafd1fce88b865efd81baafe35b2742ac4f6f536335317823adda8e30/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/f410814d2c3a1f8a42712ff22ad70f19c44b096cc057d30e5a22e7ca2862f7ac/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/110bdbe14e302287e09772e350e576bf1799b55649285e9c52c73942d6763057/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/14253773b30aec91847edba96ad1b3f35ec82e96bd40b95ed2dc326057099b65/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/6b5ca44ac970dc4176b346ba739d27d0082a23c3a714cf4caa7bc524f211c0bf/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/0351d9307b651a793fb84cbc97a2189e8e9370fc4ead79f78e08a44337d8da76/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/a255580601ea365c95b51351d2aa0f78ca768e241b3c7f4384be46644bd03c41/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/d46a7a4fa9a292db5ba6a5b785e074cf201f710dd368eee49e68c4bdae8dd2cd/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/3c464b25699c74ec47e0a10617ee34bdb203d48cdf3bfaaea4bef34ef32eba22/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/9538865356c5860e861f9a415b095f9dfbb8bdf2bf5fd2e49784e16ebe30d77e/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/071ecb7c6805d9fd18cc66bf5be5cae75fbf0bb9ae21c6c9cff78cbb3cd22459/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/bc7ac9f2b3d298d81fa59f93650f23b2290382ae7b187e34f85c7c51f316d395/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/ea70f4ffb1f0f0480ab0c3583444d0c64746865f667e67c035820c26697a2b05/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/217e8b3c029a9f2217936ce09492a37a051f3a2485007d38d1fad954fe3a1636/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/2716946959cd8e52acf5c30bfdd41ad6e5dfef0b1b1f699ef692e35afd83d762/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/f49ee409c48eb8ac903dbcd307d71f7383c4643f68a96649ed09a0ea34483b44/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/9138e49953e19be79084e83d6694e3d881d449e991abb210fe85bc495d4f2e8f/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/d1f363835bd0348f1d5f7e8537405996db67dabcfacfe6776f2d1ff93d252437/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/4324f08ce2700186b275a426d24667365a3c223915ddf10ac950252c150ad4d9/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/65a881ed2aca89a497efdc048e05a855fbfb22dd95d4c6284f1bbacce6f9983e/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/5fc64e21911cefb7b46a835f0135592a5565140be8f6a64ea36b354a91902722/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/11ab20ea4fbf9e8af1f88551e0cc2f074b448e7031ae51097659863ea8a43f77/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/c15c3216ee729b637f4bdbb9894186b6d29d2198fc9dc2d7bd16aec6c6cee721/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/2690f1436f2d624492fece8c30852880bbe80639e5d87afa117dc2ea23c61f68/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/b74110d404bf73407d50de3f8a060f25ba619e03cf21fd67397f7a144312e12b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/91cd211baf08acadf50d20ee9530fac2eca223cbb0e46f3a5eadf6c5fd983f58/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2acf56b2dafb084565398d792a485a0d571e1b1eedda58f35704b805869d5909/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/567a2243dca3e3454cc1f58c391882748d6825f03bfbb2c9f4f8daa638c1e15c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/419360c65e9b78f63adb6de6656f410c53ac0c44ddc42db405765ef0bc0cca94/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/b5dd32e783603595a29258d7a7a5678789d850b6c641e314801350f222804507/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/28799d9d37f07a6d6b9bc8c4d1a2dd6311892c6139dafb79e653ba53c05fddb3/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/445fb21b2cba2657efec616d3971114ae45a0b3ffd786af402f2a172ce7eee00/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f8849a1286a23cb975993af44731eae01277a09185ddb1da44177c4bb7330346/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b82e41314c8b84f2ac07855915b8e85b8c4d7721abd263d61f37c6acd43a7dcf/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:47:03.734673 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.734277    5503 manager.go:217] Machine: {Timestamp:2026-10-30 21:47:03.733811056 +0000 UTC m=+0.068387817 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099063 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec295915e20b70bed663a35ef82cfb06 SystemUUID:ec295915-e20b-70be-d663-a35ef82cfb06 BootID:5641d39c-9cc5-45f2-ac9e-51c7179c70a0 Filesystems:[{Device:/var/lib/kubelet/pods/3d267e8e-2220-4d18-a54e-9290c579b7bf/volumes/kubernetes.io~projected/kube-api-access-hl95m DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b021c8932708178d0aeb8deb35c1c975b3762ad9c5f92d1e4b55cfc0a379038/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:301 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67694dfd-17df-4a2a-b598-f03003ee0757/volumes/kubernetes.io~projected/kube-api-access-tv67g DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62848352f228b49c54a4f4ea85da47a7585e595e04d7df32d381ab0bc5e5290b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~projected/kube-api-access-4m925 DeviceMajor:0 DeviceMinor:252 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173/volumes/kubernetes.io~projected/kube-api-access-wpn52 DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbbe7220-365b-45da-b66c-9cb581843a2e/volumes/kubernetes.io~projected/kube-api-access-wgtr8 DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~projected/kube-api-access-62nzc DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f/volumes/kubernetes.io~projected/kube-api-access-9x78p DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40ff8e8394affe59eda38a15264189ca68e7106421637d44b77b26adcae21055/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5231c439950f7a74ff3c151c13a8dd9eda810216d4d0ea7a59bc9581a2efd218/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:298 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61248224-57d8-4277-9f87-a0d49e8c4c6f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:330 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/61248224-57d8-4277-9f87-a0d49e8c4c6f/volumes/kubernetes.io~projected/kube-api-access-qq27q DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55d27993-063d-499a-b3e8-ef6b2d0ea03e/volumes/kubernetes.io~projected/kube-api-access-p27bh DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5ad80ae3125aa77f85e1770becaccdd2d70c534715ddc2e2021c705115f48bc/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~projected/kube-api-access-jbrcq DeviceMajor:0 DeviceMinor:304 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14e12922-2bce-41b4-935d-5d3708d1694e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:336 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5217083d967451f93b8fa6fb8907954480a96f9eccc72a7f1b94d282aee81769/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:303 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/709a6375f3498ce1d2d6c1a9d930ca95952d13b3bfab9948d16e7f4717d9c784/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:298 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b31df362-1dd4-410c-a946-545f9146c7a6/volumes/kubernetes.io~projected/kube-api-access-bzp8n DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3151d50d8558f4ceb95b6990e4567bc39ba6d8b3e7dbb249a76be4eb098b98d6/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:248 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c388501b-fe35-4373-ba59-7391b3c79fb5/volumes/kubernetes.io~projected/kube-api-access-ntqct DeviceMajor:0 DeviceMinor:285 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8748e87c-9dcc-442b-b739-820276e15f03/volumes/kubernetes.io~projected/kube-api-access-cnnxw DeviceMajor:0 DeviceMinor:342 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb2cb86387fd524bfca0708dbd67a51a4538362810bfdfcb1e2fe4668d23e1f0/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~projected/kube-api-access-8vgp7 DeviceMajor:0 DeviceMinor:247 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8721e15598b631db1c447a28c36a6e53c104e7c059ecfb77767f07676e98c78/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:295 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:301 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:249 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0dcde6fd21a29f326adb968b10cebfff659dd095a09112046b2d824191baea3a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91a3fbfd-209a-4c30-8fb9-9e4812d02f27/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d964f836bc055f3e1d57d522711eb61c5f75fdc7a64d6c1aca0aaf6c1d137d0/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:302 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94c5e9d17c38886cd2d808b1d43e91c1bf6a8a1223923f06297e6b753267a526/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/620dcfadd5bd635eebe318ec7c4fe8489c98987722eea8fded7dc638698fc386/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:301 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:300 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbbe7220-365b-45da-b66c-9cb581843a2e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:297 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e620885a-79e5-483a-9276-e644afb3f27b/volumes/kubernetes.io~projected/kube-api-access-rbplt DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e5c7e5a7-6225-4513-a2cc-592dfe01d018/volumes/kubernetes.io~projected/kube-api-access-h4wcq DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d267e8e-2220-4d18-a54e-9290c579b7bf/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:301 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:296 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4493288d0421badd9f71c2f9cf5cb5390c1fde31ab9d24333711dd6bee04c63/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bac3b80e-3555-4a21-a9bd-4dec15bcbaed/volumes/kubernetes.io~projected/kube-api-access-5gtlx DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5439ee4318478fa34ff8c2b85c119b0a03d30e3712a1465b4c5516a3864c3b6/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4a8ef7093b01236729bcf699b92f5025a91ee35c7416bcfabe1170f65612fbb9/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:299 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8748e87c-9dcc-442b-b739-820276e15f03/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:349 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/584a94f52b49f6086fa12e2e932e35d3d05ca98125fbf6419796b31c48e112b4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d87bf1f4-0cf2-4c1d-a513-2c6862b68970/volumes/kubernetes.io~projected/kube-api-access-69cqv DeviceMajor:0 DeviceMinor:286 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fee2c691-31c8-4fb6-9e9f-687116de2537/volumes/kubernetes.io~projected/kube-api-access-lt9qk DeviceMajor:0 DeviceMinor:340 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:40ff8e8394affe5 MacAddress:62:2a:70:3e:26:6a Speed:10000 Mtu:8901} {Name:4a8ef7093b01236 MacAddress:52:c6:72:3a:88:10 Speed:10000 Mtu:8901} {Name:5217083d967451f MacAddress:16:78:6d:38:d0:b1 Speed:10000 Mtu:8901} {Name:5829e5e450598e9 MacAddress:7e:4b:d1:9e:3d:58 Speed:10000 Mtu:8901} {Name:620dcfadd5bd635 MacAddress:5e:0f:cd:63:be:b1 Speed:10000 Mtu:8901} {Name:6b021c893270817 MacAddress:56:5b:fb:4c:0e:e1 Speed:10000 Mtu:8901} {Name:94c5e9d17c38886 MacAddress:b6:0d:1f:ba:86:71 Speed:10000 Mtu:8901} {Name:b8721e15598b631 MacAddress:76:46:77:a1:c3:1e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c3:c6:64:6f Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:e4:25:54:92:8a Speed:0 Mtu:8901} {Name:c9b7b5da1818938 MacAddress:86:21:b9:f0:f9:d5 Speed:10000 Mtu:8901} {Name:cb2cb86387fd524 MacAddress:ca:a5:c4:e1:38:e1 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c3:c6:64:6f Speed:0 Mtu:9001} {Name:f4493288d0421ba MacAddress:c6:2c:38:0d:1c:c9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:46:ae:31:eb:db:0b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:0e:43:ab:77:02:40 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:47:03.734901 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.734678    5503 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:47:03.734901 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.734730    5503 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:47:03.735081 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735050    5503 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:47:03.735339 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735082    5503 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-86.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:47:03.735390 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735343    5503 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:47:03.735390 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735353    5503 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:47:03.735390 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735366    5503 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:47:03.735390 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735386    5503 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:47:03.735514 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735506    5503 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:47:03.735715 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735705    5503 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:47:03.735833 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735824    5503 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:47:03.735868 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735860    5503 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:47:03.735897 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735872    5503 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:47:03.735897 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735883    5503 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:47:03.735897 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735892    5503 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:47:03.735989 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.735900    5503 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:47:03.736565 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.736548    5503 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:47:03.736565 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.736560    5503 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:47:03.737129 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.737113    5503 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:47:03.739978 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.739960    5503 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:47:03.739978 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.739976    5503 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:47:03.740093 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740077    5503 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:47:03.740278 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740267    5503 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:47:03.740313 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740296    5503 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:47:03.740392 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740383    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:47:03.740422 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740398    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:47:03.740422 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740404    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:47:03.740422 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740412    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:47:03.740422 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740421    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:47:03.740523 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740430    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:47:03.740523 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740440    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:47:03.740523 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740447    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:47:03.740523 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740460    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:47:03.740523 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740467    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:47:03.740523 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740486    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:47:03.740523 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740501    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:47:03.740801 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740527    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:47:03.740801 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.740535    5503 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:47:03.744467 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.744193    5503 server.go:1263] "Started kubelet"
Oct 30 21:47:03.744825 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.744591    5503 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:47:03.745085 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.745022    5503 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:47:03.745153 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.745104    5503 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:47:03.746435 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.745906    5503 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:47:03.745939 ip-10-0-141-86 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:47:03.748939 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.748910    5503 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.749814    5503 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.751023    5503 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.751683    5503 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.751695    5503 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.751785    5503 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:40:53 +0000 UTC" deadline="2028-05-17 03:02:02.367370583 +0000 UTC"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.751825    5503 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13541h14m58.615548816s"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:03.751943    5503 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:47:03.753155 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.752345    5503 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:47:03.755935 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.755912    5503 factory.go:55] Registering systemd factory
Oct 30 21:47:03.756050 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.755939    5503 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756288    5503 factory.go:153] Registering CRI-O factory
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756326    5503 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756376    5503 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756396    5503 factory.go:103] Registering Raw factory
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756408    5503 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756521    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert" seLinuxMountContext=""
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756572    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756586    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87bf1f4-0cf2-4c1d-a513-2c6862b68970" volumeName="kubernetes.io/empty-dir/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-volume-directive-shadow" seLinuxMountContext=""
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756611    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87bf1f4-0cf2-4c1d-a513-2c6862b68970" volumeName="kubernetes.io/projected/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-api-access-69cqv" seLinuxMountContext=""
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756627    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24" volumeName="kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-tuned" seLinuxMountContext=""
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756653    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config" seLinuxMountContext=""
Oct 30 21:47:03.756676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756674    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173" volumeName="kubernetes.io/secret/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-cert" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756693    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756710    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756731    5503 manager.go:319] Starting recovery of all containers
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756732    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67" volumeName="kubernetes.io/configmap/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756745    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173" volumeName="kubernetes.io/projected/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-kube-api-access-wpn52" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756756    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d267e8e-2220-4d18-a54e-9290c579b7bf" volumeName="kubernetes.io/secret/3d267e8e-2220-4d18-a54e-9290c579b7bf-metrics-tls" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756775    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756788    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756805    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" volumeName="kubernetes.io/projected/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-kube-api-access-5gtlx" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756825    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbe7220-365b-45da-b66c-9cb581843a2e" volumeName="kubernetes.io/projected/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-api-access-wgtr8" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756839    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c388501b-fe35-4373-ba59-7391b3c79fb5" volumeName="kubernetes.io/empty-dir/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-textfile" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756852    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c388501b-fe35-4373-ba59-7391b3c79fb5" volumeName="kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-tls" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756871    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c7e5a7-6225-4513-a2cc-592dfe01d018" volumeName="kubernetes.io/secret/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756885    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c388501b-fe35-4373-ba59-7391b3c79fb5" volumeName="kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756896    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e620885a-79e5-483a-9276-e644afb3f27b" volumeName="kubernetes.io/projected/e620885a-79e5-483a-9276-e644afb3f27b-kube-api-access-rbplt" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756915    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fee2c691-31c8-4fb6-9e9f-687116de2537" volumeName="kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756940    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a3fbfd-209a-4c30-8fb9-9e4812d02f27" volumeName="kubernetes.io/configmap/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756964    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.756982    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87bf1f4-0cf2-4c1d-a513-2c6862b68970" volumeName="kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757004    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87bf1f4-0cf2-4c1d-a513-2c6862b68970" volumeName="kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757017    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f" volumeName="kubernetes.io/projected/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-kube-api-access-9x78p" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757037    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d267e8e-2220-4d18-a54e-9290c579b7bf" volumeName="kubernetes.io/empty-dir/3d267e8e-2220-4d18-a54e-9290c579b7bf-tmp-dir" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757055    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d267e8e-2220-4d18-a54e-9290c579b7bf" volumeName="kubernetes.io/projected/3d267e8e-2220-4d18-a54e-9290c579b7bf-kube-api-access-hl95m" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757068    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8748e87c-9dcc-442b-b739-820276e15f03" volumeName="kubernetes.io/secret/8748e87c-9dcc-442b-b739-820276e15f03-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757091    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a3fbfd-209a-4c30-8fb9-9e4812d02f27" volumeName="kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-tls" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757104    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757115    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f" volumeName="kubernetes.io/configmap/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-serviceca" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757136    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55d27993-063d-499a-b3e8-ef6b2d0ea03e" volumeName="kubernetes.io/configmap/55d27993-063d-499a-b3e8-ef6b2d0ea03e-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757157    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757169    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c7e5a7-6225-4513-a2cc-592dfe01d018" volumeName="kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-env-overrides" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757197    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e620885a-79e5-483a-9276-e644afb3f27b" volumeName="kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757211    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a3fbfd-209a-4c30-8fb9-9e4812d02f27" volumeName="kubernetes.io/projected/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-kube-api-access-8vgp7" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757223    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config" seLinuxMountContext=""
Oct 30 21:47:03.757251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757255    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c7e5a7-6225-4513-a2cc-592dfe01d018" volumeName="kubernetes.io/projected/e5c7e5a7-6225-4513-a2cc-592dfe01d018-kube-api-access-h4wcq" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757269    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d267e8e-2220-4d18-a54e-9290c579b7bf" volumeName="kubernetes.io/configmap/3d267e8e-2220-4d18-a54e-9290c579b7bf-config-volume" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757307    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61248224-57d8-4277-9f87-a0d49e8c4c6f" volumeName="kubernetes.io/projected/61248224-57d8-4277-9f87-a0d49e8c4c6f-kube-api-access-qq27q" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757329    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" volumeName="kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757343    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c388501b-fe35-4373-ba59-7391b3c79fb5" volumeName="kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757355    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87bf1f4-0cf2-4c1d-a513-2c6862b68970" volumeName="kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757375    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e620885a-79e5-483a-9276-e644afb3f27b" volumeName="kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757388    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757401    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a3fbfd-209a-4c30-8fb9-9e4812d02f27" volumeName="kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757439    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" volumeName="kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757452    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbe7220-365b-45da-b66c-9cb581843a2e" volumeName="kubernetes.io/secret/bbbe7220-365b-45da-b66c-9cb581843a2e-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757465    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61248224-57d8-4277-9f87-a0d49e8c4c6f" volumeName="kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757488    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67694dfd-17df-4a2a-b598-f03003ee0757" volumeName="kubernetes.io/empty-dir/67694dfd-17df-4a2a-b598-f03003ee0757-tmp-dir" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757502    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757515    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbe7220-365b-45da-b66c-9cb581843a2e" volumeName="kubernetes.io/configmap/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757546    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87bf1f4-0cf2-4c1d-a513-2c6862b68970" volumeName="kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-metrics-client-ca" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757559    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c7e5a7-6225-4513-a2cc-592dfe01d018" volumeName="kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-config" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757597    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24" volumeName="kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-tmp" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757612    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757627    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757677    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757694    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67" volumeName="kubernetes.io/secret/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-agent-certs" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757708    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e12922-2bce-41b4-935d-5d3708d1694e" volumeName="kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757720    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757735    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757747    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c388501b-fe35-4373-ba59-7391b3c79fb5" volumeName="kubernetes.io/projected/c388501b-fe35-4373-ba59-7391b3c79fb5-kube-api-access-ntqct" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757760    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c388501b-fe35-4373-ba59-7391b3c79fb5" volumeName="kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757772    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e5c7e5a7-6225-4513-a2cc-592dfe01d018" volumeName="kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757785    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55d27993-063d-499a-b3e8-ef6b2d0ea03e" volumeName="kubernetes.io/projected/55d27993-063d-499a-b3e8-ef6b2d0ea03e-kube-api-access-p27bh" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757798    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67694dfd-17df-4a2a-b598-f03003ee0757" volumeName="kubernetes.io/projected/67694dfd-17df-4a2a-b598-f03003ee0757-kube-api-access-tv67g" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757811    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8748e87c-9dcc-442b-b739-820276e15f03" volumeName="kubernetes.io/projected/8748e87c-9dcc-442b-b739-820276e15f03-kube-api-access-cnnxw" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757822    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc41423-0f20-491b-b9ee-fff3583ca471" volumeName="kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757833    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b31df362-1dd4-410c-a946-545f9146c7a6" volumeName="kubernetes.io/projected/b31df362-1dd4-410c-a946-545f9146c7a6-kube-api-access-bzp8n" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757844    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" volumeName="kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757857    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbbe7220-365b-45da-b66c-9cb581843a2e" volumeName="kubernetes.io/empty-dir/bbbe7220-365b-45da-b66c-9cb581843a2e-data-volume" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757868    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757882    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f278f63-40f1-40eb-8d8d-04f297071c1f" volumeName="kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757895    5503 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24" volumeName="kubernetes.io/projected/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-kube-api-access-62nzc" seLinuxMountContext=""
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757907    5503 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:47:03.759013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.757914    5503 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:47:03.778216 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.778173    5503 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:47:03.781342 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.781318    5503 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:47:03.781458 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.781347    5503 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:47:03.781458 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.781368    5503 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:47:03.781458 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:03.781425    5503 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:47:03.791435 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.791413    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:47:03.791594 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.791576    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:47:03.804710 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.804681    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e" exitCode=0
Oct 30 21:47:03.811884 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.811854    5503 generic.go:356] "Generic (PLEG): container finished" podID="c388501b-fe35-4373-ba59-7391b3c79fb5" containerID="7bced63265738d8a76cc754c697d75a6bfe2da04fa85068ee2fd061696360be7" exitCode=0
Oct 30 21:47:03.821359 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.821337    5503 generic.go:356] "Generic (PLEG): container finished" podID="1c08c33d0fce1d0a09d0e1c911723af2" containerID="cd4ac741cdee188b0feac0fb96c093bcdc5da49d4e150bbfcfbbac363971b09c" exitCode=0
Oct 30 21:47:03.829293 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.829266    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:47:03.829690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.829663    5503 generic.go:356] "Generic (PLEG): container finished" podID="e5c7e5a7-6225-4513-a2cc-592dfe01d018" containerID="2db7728b4e9cf44fcc8885b0c0211f185117d77fe82e5ddf916ad972c35e5978" exitCode=1
Oct 30 21:47:03.836916 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.836894    5503 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="ee3353c39b8ba16b498770738c29f41e02df2c3827aa27480d69e409fc0cb9f8" exitCode=0
Oct 30 21:47:03.837019 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.836919    5503 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="628d9c9b9ba0e9cf66b9c106985dd17ac92e2a19d841e448479aa94342766386" exitCode=0
Oct 30 21:47:03.837019 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.836930    5503 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="85d534889ae0ab84d4e2fbe01645fab86c085a51b8116f8448ff6b9781c16513" exitCode=0
Oct 30 21:47:03.837019 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.836936    5503 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="f3b12fc83158184fa6e5d366b6e3817a9b3b740f643a671b0b581ad698d80bf8" exitCode=0
Oct 30 21:47:03.837019 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.836942    5503 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="68de7a039db3f43bbc062a93d4f1375a398907ffc4685f71c6b3ca1616022c4c" exitCode=0
Oct 30 21:47:03.837019 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.836948    5503 generic.go:356] "Generic (PLEG): container finished" podID="bac3b80e-3555-4a21-a9bd-4dec15bcbaed" containerID="694b5606965af93303de405d57b2e8e3e4e1478c9b9d1594febad56a57e574f3" exitCode=0
Oct 30 21:47:03.852027 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:03.852006    5503 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:47:03.861348 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.861328    5503 manager.go:324] Recovery completed
Oct 30 21:47:03.880832 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.880809    5503 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:47:03.881909 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:03.881888    5503 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:47:03.882774 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.882762    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:47:03.882818 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.882789    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:47:03.882818 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.882800    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:47:03.885820 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.885800    5503 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:47:03.885820 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.885816    5503 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:47:03.885974 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.885816    5503 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:47:03.885974 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.885888    5503 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:47:03.886054 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886013    5503 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:47:03.886054 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886025    5503 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:47:03.886111 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886096    5503 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:47:03.886111 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886102    5503 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:47:03.886111 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886108    5503 policy_none.go:50] "Start"
Oct 30 21:47:03.886202 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886120    5503 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:47:03.886202 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886130    5503 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:47:03.886260 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886248    5503 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:47:03.886290 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886260    5503 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:47:03.886290 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886266    5503 policy_none.go:44] "Start"
Oct 30 21:47:03.886714 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886696    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:47:03.886785 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886730    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:47:03.886785 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.886744    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:47:03.890827 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.890798    5503 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:47:03.893606 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.893591    5503 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:47:03.893716 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:03.893623    5503 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:47:03.893716 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.893648    5503 server.go:87] "Starting device plugin registration server"
Oct 30 21:47:03.893969 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.893955    5503 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:47:03.894028 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.893970    5503 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:47:03.894086 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.894049    5503 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:47:03.894182 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.894148    5503 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:47:03.894234 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.894195    5503 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:47:03.894234 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.894202    5503 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:47:03.894336 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.894261    5503 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:47:03.894181078Z","UUID":"815ffbe0-7ee6-4890-ac4c-9787aef06151","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:47:03.895763 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.895745    5503 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:47:03.895854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.895773    5503 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:47:03.896735 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:03.896715    5503 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:47:03.896813 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:03.896757    5503 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-86.us-west-2.compute.internal\" not found"
Oct 30 21:47:03.939006 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.938970    5503 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:47:03.939006 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.938970    5503 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:47:03.939215 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.939159    5503 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:47:03.945028 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.945009    5503 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:47:03.994294 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.994221    5503 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:47:03.995259 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.995241    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:47:03.995358 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.995300    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:47:03.995358 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.995312    5503 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-86.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:47:03.995358 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:03.995351    5503 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.032954 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.032917    5503 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:47:04.082317 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.082265    5503 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"]
Oct 30 21:47:04.083069 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.083012    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"25a4aaee5f241beed648c981baba69bb","Type":"ContainerStarted","Data":"12aae623aa5f03893f4c73ece2efea390f20bd454edca06e8ed3785e0556d9dc"}
Oct 30 21:47:04.083188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.083080    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"25a4aaee5f241beed648c981baba69bb","Type":"ContainerStarted","Data":"11f17832001b7fcb34780cd1c93b8887186f6e7013df2ee85084985e8b4f2287"}
Oct 30 21:47:04.083188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.083088    5503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.083188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.083153    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"1c08c33d0fce1d0a09d0e1c911723af2","Type":"ContainerStarted","Data":"366f88a98a53c1ad12dc021ed420299fc2995e49558d18ef9c173db288baf8aa"}
Oct 30 21:47:04.083188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.083167    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"1c08c33d0fce1d0a09d0e1c911723af2","Type":"ContainerDied","Data":"cd4ac741cdee188b0feac0fb96c093bcdc5da49d4e150bbfcfbbac363971b09c"}
Oct 30 21:47:04.083188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.083088    5503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.083188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.083181    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal" event={"ID":"1c08c33d0fce1d0a09d0e1c911723af2","Type":"ContainerStarted","Data":"25f208908b55590502f3f9101bbc154dd723c0bbaa66998ce154a3c618400309"}
Oct 30 21:47:04.114598 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.114571    5503 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.114764 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.114663    5503 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.141371 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.141340    5503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:47:04.141523 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.141391    5503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.142106 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.142087    5503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:47:04.142153 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.142131    5503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.241339 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.241303    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.241498 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.241353    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.241498 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.241372    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25a4aaee5f241beed648c981baba69bb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"25a4aaee5f241beed648c981baba69bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.342184 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.342103    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.342184 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.342139    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25a4aaee5f241beed648c981baba69bb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"25a4aaee5f241beed648c981baba69bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.342184 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.342156    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.342428 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.342220    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.342428 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.342297    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c08c33d0fce1d0a09d0e1c911723af2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"1c08c33d0fce1d0a09d0e1c911723af2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.342428 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.342341    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/25a4aaee5f241beed648c981baba69bb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" (UID: \"25a4aaee5f241beed648c981baba69bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.737082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.737052    5503 apiserver.go:52] "Watching apiserver"
Oct 30 21:47:04.838174 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.837625    5503 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:47:04.839758 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.839529    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-ppjfl","openshift-ovn-kubernetes/ovnkube-node-gvxg2","openshift-dns/node-resolver-zccbp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal","openshift-image-registry/node-ca-d66x4","openshift-monitoring/node-exporter-8z59d","openshift-dns/dns-default-ld584","openshift-insights/insights-runtime-extractor-2sdgq","openshift-network-operator/iptables-alerter-ssjj9","openshift-cluster-node-tuning-operator/tuned-fpgxc","openshift-ingress-canary/ingress-canary-xrbvw","openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj","kube-system/konnectivity-agent-blztt","openshift-console/console-5f77674798-w874p","openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb","openshift-multus/network-metrics-daemon-j6kdw","openshift-network-diagnostics/network-check-target-tv4l4","kube-system/global-pull-secret-syncer-lwfgr","openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2","openshift-multus/multus-49fc7","kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal","openshift-monitoring/alertmanager-main-0"]
Oct 30 21:47:04.841512 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.841489    5503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.841664 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.841551    5503 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.852391 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.852352    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:04.854701 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.854684    5503 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:47:04.893040 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893013    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rn9hr\""
Oct 30 21:47:04.893040 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893031    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kjn6x\""
Oct 30 21:47:04.893040 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893046    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bg2hw\""
Oct 30 21:47:04.893235 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893031    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9dfpb\""
Oct 30 21:47:04.893235 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893085    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fd5qx\""
Oct 30 21:47:04.893235 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893096    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6mkvf\""
Oct 30 21:47:04.893235 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893104    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:47:04.893574 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893560    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.893692 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893680    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:47:04.893725 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893708    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.893753 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.893686    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.894129 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.894106    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:47:04.894262 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.894117    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.907137 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.907114    5503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:47:04.907278 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.907165    5503 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:47:04.907278 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.907168    5503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.907278 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.907209    5503 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-86.us-west-2.compute.internal"
Oct 30 21:47:04.944601 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944569    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-log-socket\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.944601 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944599    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.944832 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944624    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.944832 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944719    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnxw\" (UniqueName: \"kubernetes.io/projected/8748e87c-9dcc-442b-b739-820276e15f03-kube-api-access-cnnxw\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:04.944832 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.944754    5503 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:47:04.944832 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944762    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-var-lib-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.944832 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944801    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-config\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.944832 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.944827    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.444811831 +0000 UTC m=+1.779388598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"service-ca" not registered
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944852    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-cert\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944874    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-tls\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944951    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.944983    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945029    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-conf\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945060    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-multus-daemon-config\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945095    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8748e87c-9dcc-442b-b739-820276e15f03-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945120    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67694dfd-17df-4a2a-b598-f03003ee0757-tmp-dir\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945121    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945177    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqct\" (UniqueName: \"kubernetes.io/projected/c388501b-fe35-4373-ba59-7391b3c79fb5-kube-api-access-ntqct\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945185    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67694dfd-17df-4a2a-b598-f03003ee0757-tmp-dir\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945206    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945234    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.945257 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945259    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbbe7220-365b-45da-b66c-9cb581843a2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945284    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945321    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-node-log\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945347    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945397    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.945441    5503 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945453    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-system-cni-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945515    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-agent-certs\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.945557    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.445545857 +0000 UTC m=+1.780122607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945607    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-hostroot\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945653    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbbe7220-365b-45da-b66c-9cb581843a2e-crio-socket\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945681    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-multus-certs\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945708    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-tmp\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945736    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtlx\" (UniqueName: \"kubernetes.io/projected/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-kube-api-access-5gtlx\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945778    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv67g\" (UniqueName: \"kubernetes.io/projected/67694dfd-17df-4a2a-b598-f03003ee0757-kube-api-access-tv67g\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:47:04.945823 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945812    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945832    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-agent-certs\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945838    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945866    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-tmp\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945867    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945915    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtr8\" (UniqueName: \"kubernetes.io/projected/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-api-access-wgtr8\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945935    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-kubelet\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945966    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-cnibin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.945993    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d267e8e-2220-4d18-a54e-9290c579b7bf-metrics-tls\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946055    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946088    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-textfile\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946113    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-run\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946135    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-serviceca\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946156    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946170    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946170    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysconfig\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946191    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-textfile\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946214    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-multus\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946246    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq27q\" (UniqueName: \"kubernetes.io/projected/61248224-57d8-4277-9f87-a0d49e8c4c6f-kube-api-access-qq27q\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946271    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.946356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946366    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-netns\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946396    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946423    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-lib-modules\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946477    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946510    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946545    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-system-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946585    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-wtmp\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946616    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-systemd-units\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946645    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-slash\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946665    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-bin\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946690    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946717    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-host\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946749    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946762    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-dbus\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946790    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovn-node-metrics-cert\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946817    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-socket-dir-parent\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946863    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946879    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-kubernetes\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946949    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.946970    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-device-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947012    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbplt\" (UniqueName: \"kubernetes.io/projected/e620885a-79e5-483a-9276-e644afb3f27b-kube-api-access-rbplt\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.947082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947036    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947101    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947163    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55d27993-063d-499a-b3e8-ef6b2d0ea03e-iptables-alerter-script\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947197    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-konnectivity-ca\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947248    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x78p\" (UniqueName: \"kubernetes.io/projected/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-kube-api-access-9x78p\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947344    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55d27993-063d-499a-b3e8-ef6b2d0ea03e-iptables-alerter-script\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947342    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947376    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947376    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67-konnectivity-ca\") pod \"konnectivity-agent-blztt\" (UID: \"cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67\") " pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947449    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.947473    5503 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.947529    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.447518116 +0000 UTC m=+1.782094876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.947542    5503 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947554    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-k8s-cni-cncf-io\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.947585    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.447571326 +0000 UTC m=+1.782148083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-config" not registered
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947601    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d267e8e-2220-4d18-a54e-9290c579b7bf-tmp-dir\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947622    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947657    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-host\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947685    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-bin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947711    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947720    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d267e8e-2220-4d18-a54e-9290c579b7bf-tmp-dir\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947750    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947838    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-modprobe-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947873    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-systemd\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947899    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.947908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947920    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-cni-binary-copy\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.947958    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d267e8e-2220-4d18-a54e-9290c579b7bf-config-volume\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948003    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-etc-kubernetes\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948040    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948111    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgp7\" (UniqueName: \"kubernetes.io/projected/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-kube-api-access-8vgp7\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948140    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cqv\" (UniqueName: \"kubernetes.io/projected/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-api-access-69cqv\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948166    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-sys\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948193    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-kubelet\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948251    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpn52\" (UniqueName: \"kubernetes.io/projected/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-kube-api-access-wpn52\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948285    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948315    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d27993-063d-499a-b3e8-ef6b2d0ea03e-host-slash\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948341    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p27bh\" (UniqueName: \"kubernetes.io/projected/55d27993-063d-499a-b3e8-ef6b2d0ea03e-kube-api-access-p27bh\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948365    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-etc-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948388    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4wcq\" (UniqueName: \"kubernetes.io/projected/e5c7e5a7-6225-4513-a2cc-592dfe01d018-kube-api-access-h4wcq\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948405    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl95m\" (UniqueName: \"kubernetes.io/projected/3d267e8e-2220-4d18-a54e-9290c579b7bf-kube-api-access-hl95m\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948424    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948441    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-sys\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948464    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948498    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-script-lib\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948531    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948559    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.948567    5503 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948588    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62nzc\" (UniqueName: \"kubernetes.io/projected/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-kube-api-access-62nzc\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948620    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-accelerators-collector-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948664    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.948683    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.448666881 +0000 UTC m=+1.783243644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948717    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrcq\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948742    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-tuned\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948770    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-kubelet-config\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948794    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-netd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948821    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67694dfd-17df-4a2a-b598-f03003ee0757-hosts-file\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948848    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-tuned\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948866    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948894    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948924    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzp8n\" (UniqueName: \"kubernetes.io/projected/b31df362-1dd4-410c-a946-545f9146c7a6-kube-api-access-bzp8n\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:04.948949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.948957    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949021    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-env-overrides\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949124    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949168    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949187    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-root\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949203    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-metrics-client-ca\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949220    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbbe7220-365b-45da-b66c-9cb581843a2e-data-volume\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949236    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-netns\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949268    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-ovn\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949285    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.949300    5503 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:04.949338    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.44932819 +0000 UTC m=+1.783904958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949360    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949363    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bbbe7220-365b-45da-b66c-9cb581843a2e-data-volume\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949436    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949459    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949477    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cnibin\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949495    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-systemd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949510    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-conf-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949525    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-var-lib-kubelet\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949542    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.950101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.949568    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.978573 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978549    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:47:04.978573 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978553    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:47:04.978770 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978553    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:47:04.978770 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978709    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pvjtk\""
Oct 30 21:47:04.978898 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978878    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.978934 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978920    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pz9m9\""
Oct 30 21:47:04.978992 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978977    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:47:04.979041 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.978988    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:47:04.979041 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979022    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:47:04.979114 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979055    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r29xw\""
Oct 30 21:47:04.979114 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979060    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:47:04.979114 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979069    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:47:04.979114 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979067    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:47:04.979114 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979057    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:47:04.979283 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979157    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:47:04.979283 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979193    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:47:04.979283 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979213    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:47:04.979283 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979235    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:47:04.979283 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979248    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-879p5\""
Oct 30 21:47:04.979283 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979277    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5wdm\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979315    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979353    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hs26f\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979367    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqqg2\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979355    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9kwx8\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979384    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xlfcr\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979406    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979431    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x9s8l\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979500    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:47:04.979518 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979509    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4gblc\""
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979570    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qzzpd\""
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979577    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979617    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979725    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979740    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979764    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979768    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979795    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979818    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5wwct\""
Oct 30 21:47:04.979892 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979870    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:47:04.980151 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.979909    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tzv72\""
Oct 30 21:47:04.985800 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.985769    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.985886 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.985832    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bbbe7220-365b-45da-b66c-9cb581843a2e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.985886 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.985842    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.985886 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.985836    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-tls\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.985886 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.985872    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.986037 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.985896    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8748e87c-9dcc-442b-b739-820276e15f03-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:04.986037 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.985910    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-cert\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:47:04.987200 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.987143    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.987200 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.987164    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3d267e8e-2220-4d18-a54e-9290c579b7bf-metrics-tls\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:04.987200 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.987150    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:04.987335 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.987213    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovn-node-metrics-cert\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.988350 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.988331    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.988442 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.988368    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:04.989500 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.989485    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.989557 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.989533    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61248224-57d8-4277-9f87-a0d49e8c4c6f-metrics-certs\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:47:04.989557 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.989549    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.989694 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.989678    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/14e12922-2bce-41b4-935d-5d3708d1694e-original-pull-secret\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:04.991857 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.991844    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.992273 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992191    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:47:04.992273 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992206    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.992273 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992237    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.992434 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992312    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.992434 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992363    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:47:04.992578 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992565    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.992703 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992686    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.992779 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992735    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.992779 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992737    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:47:04.992779 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992762    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:47:04.992779 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992773    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:47:04.992779 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992778    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992797    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992851    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992859    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992867    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992869    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992888    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992892    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992900    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992929    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.992979    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993002    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.993042 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993005    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:47:04.993532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993092    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:47:04.993532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993126    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:47:04.993532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993178    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:47:04.993532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993271    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:47:04.993532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993311    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:47:04.993532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.993418    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:47:04.995738 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.995721    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:04.995803 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.995736    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-config\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.995803 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.995728    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-multus-daemon-config\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.997208 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.997191    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-serviceca\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:47:04.997359 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.997345    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-binary-copy\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:04.997404 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.997365    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.999074 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999051    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:04.999166 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999083    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:04.999166 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999091    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3d267e8e-2220-4d18-a54e-9290c579b7bf-config-volume\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:04.999166 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999111    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-ovnkube-script-lib\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:04.999166 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999118    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e620885a-79e5-483a-9276-e644afb3f27b-cni-binary-copy\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:04.999166 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999121    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-accelerators-collector-config\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:04.999166 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999134    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:04.999356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:04.999211    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5c7e5a7-6225-4513-a2cc-592dfe01d018-env-overrides\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.000403 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.000386    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c388501b-fe35-4373-ba59-7391b3c79fb5-metrics-client-ca\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.000465 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.000401    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.000465 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.000414    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.026143 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.026104    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.026596    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv67g\" (UniqueName: \"kubernetes.io/projected/67694dfd-17df-4a2a-b598-f03003ee0757-kube-api-access-tv67g\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.026596    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtr8\" (UniqueName: \"kubernetes.io/projected/bbbe7220-365b-45da-b66c-9cb581843a2e-kube-api-access-wgtr8\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.026657    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq27q\" (UniqueName: \"kubernetes.io/projected/61248224-57d8-4277-9f87-a0d49e8c4c6f-kube-api-access-qq27q\") pod \"network-metrics-daemon-j6kdw\" (UID: \"61248224-57d8-4277-9f87-a0d49e8c4c6f\") " pod="openshift-multus/network-metrics-daemon-j6kdw"
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.026690    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtlx\" (UniqueName: \"kubernetes.io/projected/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-kube-api-access-5gtlx\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.026727    5503 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.026732    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x78p\" (UniqueName: \"kubernetes.io/projected/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-kube-api-access-9x78p\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.026750    5503 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.026797    5503 projected.go:196] Error preparing data for projected volume kube-api-access-4m925 for pod openshift-console/console-5f77674798-w874p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.026860    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqct\" (UniqueName: \"kubernetes.io/projected/c388501b-fe35-4373-ba59-7391b3c79fb5-kube-api-access-ntqct\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.026952 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.026902    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925 podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:05.526874981 +0000 UTC m=+1.861451858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4m925" (UniqueName: "kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:05.027536 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.027120    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzp8n\" (UniqueName: \"kubernetes.io/projected/b31df362-1dd4-410c-a946-545f9146c7a6-kube-api-access-bzp8n\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.027536 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.027131    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbplt\" (UniqueName: \"kubernetes.io/projected/e620885a-79e5-483a-9276-e644afb3f27b-kube-api-access-rbplt\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.027536 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.027330    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62nzc\" (UniqueName: \"kubernetes.io/projected/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-kube-api-access-62nzc\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.027712 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.027542    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl95m\" (UniqueName: \"kubernetes.io/projected/3d267e8e-2220-4d18-a54e-9290c579b7bf-kube-api-access-hl95m\") pod \"dns-default-ld584\" (UID: \"3d267e8e-2220-4d18-a54e-9290c579b7bf\") " pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:05.028046 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.027847    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:05.028046 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.027982    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnxw\" (UniqueName: \"kubernetes.io/projected/8748e87c-9dcc-442b-b739-820276e15f03-kube-api-access-cnnxw\") pod \"monitoring-plugin-77f4f76594-2vq2h\" (UID: \"8748e87c-9dcc-442b-b739-820276e15f03\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:05.028188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.028046    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p27bh\" (UniqueName: \"kubernetes.io/projected/55d27993-063d-499a-b3e8-ef6b2d0ea03e-kube-api-access-p27bh\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:47:05.028188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.028097    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9qk\" (UniqueName: \"kubernetes.io/projected/fee2c691-31c8-4fb6-9e9f-687116de2537-kube-api-access-lt9qk\") pod \"network-check-target-tv4l4\" (UID: \"fee2c691-31c8-4fb6-9e9f-687116de2537\") " pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:47:05.028188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.028123    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cqv\" (UniqueName: \"kubernetes.io/projected/d87bf1f4-0cf2-4c1d-a513-2c6862b68970-kube-api-access-69cqv\") pod \"kube-state-metrics-5dc8d7d89b-p7fkb\" (UID: \"d87bf1f4-0cf2-4c1d-a513-2c6862b68970\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-p7fkb"
Oct 30 21:47:05.028357 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.028276    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgp7\" (UniqueName: \"kubernetes.io/projected/91a3fbfd-209a-4c30-8fb9-9e4812d02f27-kube-api-access-8vgp7\") pod \"prometheus-operator-9cd89b74c-qcvc2\" (UID: \"91a3fbfd-209a-4c30-8fb9-9e4812d02f27\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qcvc2"
Oct 30 21:47:05.028608 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.028567    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpn52\" (UniqueName: \"kubernetes.io/projected/e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173-kube-api-access-wpn52\") pod \"ingress-canary-xrbvw\" (UID: \"e0cb95c2-bb8e-4d26-8e8f-48a49ed0d173\") " pod="openshift-ingress-canary/ingress-canary-xrbvw"
Oct 30 21:47:05.028700 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.028588    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4wcq\" (UniqueName: \"kubernetes.io/projected/e5c7e5a7-6225-4513-a2cc-592dfe01d018-kube-api-access-h4wcq\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.028736 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.028700    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrcq\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\") pod \"alertmanager-main-0\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:05.051004 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.050986    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-kubelet-config\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:05.051096 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051011    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-netd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051096 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051027    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67694dfd-17df-4a2a-b598-f03003ee0757-hosts-file\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:47:05.051096 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051046    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.051096 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051062    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.051096 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051091    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-netd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051123    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-kubelet-config\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051148    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-registration-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051141    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67694dfd-17df-4a2a-b598-f03003ee0757-hosts-file\") pod \"node-resolver-zccbp\" (UID: \"67694dfd-17df-4a2a-b598-f03003ee0757\") " pod="openshift-dns/node-resolver-zccbp"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051145    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-root\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051179    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-root\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051149    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-socket-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051206    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-netns\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051242    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-netns\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051303    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-ovn\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051327    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-ovn\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051331    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051364    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cnibin\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051359    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051390    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-systemd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051406    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-cnibin\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051414    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-conf-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051438    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-var-lib-kubelet\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051473    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-systemd\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051481    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-conf-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051476    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-log-socket\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051509    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-var-lib-kubelet\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051513    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-log-socket\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051544    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051574    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-var-lib-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051592    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-conf\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051623    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051684    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051682    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-var-lib-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051709    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-node-log\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051745    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-run-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051740    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-system-cni-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051771    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-system-cni-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051779    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-conf\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051786    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-hostroot\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051794    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-node-log\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051812    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbbe7220-365b-45da-b66c-9cb581843a2e-crio-socket\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051850    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-multus-certs\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051864    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-hostroot\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051892    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bbbe7220-365b-45da-b66c-9cb581843a2e-crio-socket\") pod \"insights-runtime-extractor-2sdgq\" (UID: \"bbbe7220-365b-45da-b66c-9cb581843a2e\") " pod="openshift-insights/insights-runtime-extractor-2sdgq"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051892    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-multus-certs\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051894    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051939    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051958    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.051956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051958    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-sys-fs\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.051984    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-kubelet\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052007    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-kubelet\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052012    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-cnibin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052044    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052055    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-run\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052066    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-cnibin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052076    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052081    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052091    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-run\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052106    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysconfig\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052123    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052143    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-multus\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052165    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052183    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-netns\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052199    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-lib-modules\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052194    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-multus\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052145    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysconfig\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052222    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052255    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bac3b80e-3555-4a21-a9bd-4dec15bcbaed-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ppjfl\" (UID: \"bac3b80e-3555-4a21-a9bd-4dec15bcbaed\") " pod="openshift-multus/multus-additional-cni-plugins-ppjfl"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052259    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-netns\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052259    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-system-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052288    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-sysctl-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052299    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-wtmp\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052318    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-lib-modules\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052325    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-systemd-units\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052300    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-system-cni-dir\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052352    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-systemd-units\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052354    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-node-exporter-wtmp\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052358    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-slash\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052390    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-slash\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052406    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-bin\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052428    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-host\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052439    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-host-cni-bin\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052445    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-dbus\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052469    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-socket-dir-parent\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052474    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/14e12922-2bce-41b4-935d-5d3708d1694e-dbus\") pod \"global-pull-secret-syncer-lwfgr\" (UID: \"14e12922-2bce-41b4-935d-5d3708d1694e\") " pod="kube-system/global-pull-secret-syncer-lwfgr"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052484    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-kubernetes\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052500    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-device-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052513    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-host\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052533    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-multus-socket-dir-parent\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052536    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-k8s-cni-cncf-io\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052553    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-run-k8s-cni-cncf-io\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052542    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-kubernetes\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052566    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-host\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052583    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-bin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052599    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f-host\") pod \"node-ca-d66x4\" (UID: \"3b567cd8-76eb-4a2e-b05a-b1d8216b7c4f\") " pod="openshift-image-registry/node-ca-d66x4"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052607    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-modprobe-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052608    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b31df362-1dd4-410c-a946-545f9146c7a6-device-dir\") pod \"aws-ebs-csi-driver-node-4zxgj\" (UID: \"b31df362-1dd4-410c-a946-545f9146c7a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4zxgj"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052622    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-systemd\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052624    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-cni-bin\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052684    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-systemd\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052729    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-etc-modprobe-d\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052746    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-etc-kubernetes\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052764    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-sys\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052781    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-kubelet\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052797    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d27993-063d-499a-b3e8-ef6b2d0ea03e-host-slash\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052819    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-etc-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052844    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-sys\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052848    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55d27993-063d-499a-b3e8-ef6b2d0ea03e-host-slash\") pod \"iptables-alerter-ssjj9\" (UID: \"55d27993-063d-499a-b3e8-ef6b2d0ea03e\") " pod="openshift-network-operator/iptables-alerter-ssjj9"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052855    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-host-var-lib-kubelet\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052858    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5c7e5a7-6225-4513-a2cc-592dfe01d018-etc-openvswitch\") pod \"ovnkube-node-gvxg2\" (UID: \"e5c7e5a7-6225-4513-a2cc-592dfe01d018\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052817    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e620885a-79e5-483a-9276-e644afb3f27b-etc-kubernetes\") pod \"multus-49fc7\" (UID: \"e620885a-79e5-483a-9276-e644afb3f27b\") " pod="openshift-multus/multus-49fc7"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052844    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24-sys\") pod \"tuned-fpgxc\" (UID: \"e99ce2f2-ade1-4d4d-b8d4-a483c4e12f24\") " pod="openshift-cluster-node-tuning-operator/tuned-fpgxc"
Oct 30 21:47:05.053181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.052876    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c388501b-fe35-4373-ba59-7391b3c79fb5-sys\") pod \"node-exporter-8z59d\" (UID: \"c388501b-fe35-4373-ba59-7391b3c79fb5\") " pod="openshift-monitoring/node-exporter-8z59d"
Oct 30 21:47:05.456761 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.456724    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:05.456943 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.456814    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:05.456943 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.456842    5503 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:47:05.456943 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.456880    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:05.456943 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.456905    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:05.456943 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.456927    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:06.456909045 +0000 UTC m=+2.791485795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"service-ca" not registered
Oct 30 21:47:05.456943 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.456941    5503 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.456953    5503 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.456983    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:06.456972013 +0000 UTC m=+2.791548780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.457002    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.457013    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:06.456998671 +0000 UTC m=+2.791575432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.456953    5503 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.457063    5503 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.457066    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.457087    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:06.457081126 +0000 UTC m=+2.791657931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.457102    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:06.457095445 +0000 UTC m=+2.791672191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-config" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.457174    5503 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:05.457268 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.457219    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:06.457206767 +0000 UTC m=+2.791783526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:05.558309 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.558274    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:05.558453 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.558430    5503 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:47:05.558453 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.558452    5503 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:47:05.558513 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.558461    5503 projected.go:196] Error preparing data for projected volume kube-api-access-4m925 for pod openshift-console/console-5f77674798-w874p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:05.558544 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:05.558512    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925 podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:06.558496664 +0000 UTC m=+2.893073412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m925" (UniqueName: "kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:05.949749 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.949717    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:05.951833 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:05.951813    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:06.117662 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.117607    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:06.466337 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.466299    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:06.466514 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.466345    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:06.466514 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.466385    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:06.466514 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.466418    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:06.466514 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466437    5503 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:06.466514 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.466451    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:06.466514 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466474    5503 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:06.466514 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466510    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:08.466492562 +0000 UTC m=+4.801069309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466523    5503 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466534    5503 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.466537    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466567    5503 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466530    5503 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466597    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:08.466579216 +0000 UTC m=+4.801155981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"service-ca" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466713    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:08.466699392 +0000 UTC m=+4.801276143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466730    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:08.466721856 +0000 UTC m=+4.801298610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466751    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:08.466742179 +0000 UTC m=+4.801318931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-config" not registered
Oct 30 21:47:06.466894 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.466769    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:08.466760786 +0000 UTC m=+4.801337539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:06.544421 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.544385    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:06.548268 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.548243    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-2vq2h"
Oct 30 21:47:06.567076 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.567039    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:06.567304 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.567283    5503 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:47:06.567377 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.567308    5503 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:47:06.567377 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.567317    5503 projected.go:196] Error preparing data for projected volume kube-api-access-4m925 for pod openshift-console/console-5f77674798-w874p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:06.567377 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:06.567375    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925 podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:08.567357871 +0000 UTC m=+4.901934621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m925" (UniqueName: "kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:06.744859 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.744794    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ld584"
Oct 30 21:47:06.887290 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:06.887258    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:47:08.471716 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.471672    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:08.487909 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.487875    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:08.487909 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.487912    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:08.488100 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.487955    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:08.488100 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.487999    5503 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:47:08.488100 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488012    5503 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:08.488100 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.488000    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:08.488100 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488068    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:12.488050144 +0000 UTC m=+8.822626911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-config" not registered
Oct 30 21:47:08.488100 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488071    5503 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:08.488100 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488094    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:12.488080002 +0000 UTC m=+8.822656753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488131    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:12.488102872 +0000 UTC m=+8.822679622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488136    5503 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.488158    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488180    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:12.488168368 +0000 UTC m=+8.822745133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488198    5503 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.488228    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488302    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:12.488290817 +0000 UTC m=+8.822867598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"service-ca" not registered
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488301    5503 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:08.488445 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.488344    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:12.488335976 +0000 UTC m=+8.822912728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:08.589193 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.589154    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:08.589371 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.589349    5503 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:47:08.589419 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.589371    5503 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:47:08.589419 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.589382    5503 projected.go:196] Error preparing data for projected volume kube-api-access-4m925 for pod openshift-console/console-5f77674798-w874p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:08.589537 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:08.589433    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925 podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:12.58942095 +0000 UTC m=+8.923997698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m925" (UniqueName: "kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:08.687362 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:08.687322    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:09.283244 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:09.283191    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:09.466856 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:09.466826    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:12.529971 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.529928    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.529985    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.530011    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.530036    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.530077    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.530096    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530095    5503 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530128    5503 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530140    5503 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530196    5503 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530211    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:20.530187739 +0000 UTC m=+16.864764503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530235    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:20.530222525 +0000 UTC m=+16.864799276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-config" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530237    5503 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530258    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:20.530244831 +0000 UTC m=+16.864821578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530274    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:20.530268487 +0000 UTC m=+16.864845233 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530312    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:20.530297881 +0000 UTC m=+16.864874631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530192    5503 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:47:12.530464 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.530364    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:20.530349906 +0000 UTC m=+16.864926670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"service-ca" not registered
Oct 30 21:47:12.631322 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.631284    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:12.631512 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.631491    5503 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:47:12.631554 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.631517    5503 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:47:12.631554 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.631528    5503 projected.go:196] Error preparing data for projected volume kube-api-access-4m925 for pod openshift-console/console-5f77674798-w874p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:12.631624 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:12.631592    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925 podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:20.631578501 +0000 UTC m=+16.966155248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m925" (UniqueName: "kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:12.817383 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.817287    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:12.989478 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:12.989443    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:13.518401 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:13.518369    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:13.741258 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:13.741229    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:14.073912 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:14.073885    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:14.074116 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:14.074105    5503 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:47:16.872553 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:16.872520    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:17.018581 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:17.018551    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:20.083876 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.083846    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-blztt"
Oct 30 21:47:20.604750 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.604713    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:20.604925 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.604771    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:20.604925 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.604809    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:20.604925 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.604846    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:20.604925 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.604865    5503 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:20.604925 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.604903    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.604920    5503 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.604948    5503 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.604932    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:36.60491364 +0000 UTC m=+32.939490406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.604987    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:36.604971761 +0000 UTC m=+32.939548513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"service-ca" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.604954    5503 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.605018    5503 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.605034    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:36.605024378 +0000 UTC m=+32.939601130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.605049    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:36.605041246 +0000 UTC m=+32.939617993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"console-config" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.605065    5503 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.605014    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.605093    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:36.605084055 +0000 UTC m=+32.939660807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 21:47:20.605246 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.605122    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:36.605105328 +0000 UTC m=+32.939682081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 21:47:20.705916 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:20.705883    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"console-5f77674798-w874p\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") " pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:20.706044 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.706029    5503 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 21:47:20.706084 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.706050    5503 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 21:47:20.706084 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.706060    5503 projected.go:196] Error preparing data for projected volume kube-api-access-4m925 for pod openshift-console/console-5f77674798-w874p: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:20.706147 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:20.706106    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925 podName:9f278f63-40f1-40eb-8d8d-04f297071c1f nodeName:}" failed. No retries permitted until 2026-10-30 21:47:36.706094631 +0000 UTC m=+33.040671382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4m925" (UniqueName: "kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925") pod "console-5f77674798-w874p" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 21:47:25.858820 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:25.858781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:27.790725 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:27.790695    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:27.950365 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:27.950318    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:28.097393 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:28.097303    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:47:29.866148 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:29.866109    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f77674798-w874p" podUID="9f278f63-40f1-40eb-8d8d-04f297071c1f" containerName="console" containerID="cri-o://c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc" gracePeriod=15
Oct 30 21:47:30.111772 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.111748    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:30.293680 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293645    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.293854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293696    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.293854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293730    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.293854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293758    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.293854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293786    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.293854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293817    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.294102 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293967    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.294102 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294030    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") pod \"9f278f63-40f1-40eb-8d8d-04f297071c1f\" (UID: \"9f278f63-40f1-40eb-8d8d-04f297071c1f\") "
Oct 30 21:47:30.294102 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.293967    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:30.294265 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294206    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:30.294265 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294089    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:30.294656 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294278    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:30.294656 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294406    5503 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f278f63-40f1-40eb-8d8d-04f297071c1f-tmp\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.294656 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294426    5503 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.294656 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294440    5503 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-config\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.294656 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.294491    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:30.296309 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.296280    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:30.296516 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.296500    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:30.296557 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.296542    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925" pod "9f278f63-40f1-40eb-8d8d-04f297071c1f" (UID: "9f278f63-40f1-40eb-8d8d-04f297071c1f"). InnerVolumeSpecName "kube-api-access-4m925". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:30.394811 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.394775    5503 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-serving-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.394811 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.394808    5503 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-oauth-serving-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.394811 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.394819    5503 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9f278f63-40f1-40eb-8d8d-04f297071c1f-console-oauth-config\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.394968 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.394828    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m925\" (UniqueName: \"kubernetes.io/projected/9f278f63-40f1-40eb-8d8d-04f297071c1f-kube-api-access-4m925\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.394968 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.394837    5503 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f278f63-40f1-40eb-8d8d-04f297071c1f-trusted-ca-bundle\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:30.405627 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.405601    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:30.927995 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.927964    5503 generic.go:356] "Generic (PLEG): container finished" podID="9f278f63-40f1-40eb-8d8d-04f297071c1f" containerID="c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc" exitCode=0
Oct 30 21:47:30.928397 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.928020    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f77674798-w874p"
Oct 30 21:47:30.928397 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.928035    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f77674798-w874p" event={"ID":"9f278f63-40f1-40eb-8d8d-04f297071c1f","Type":"ContainerDied","Data":"c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc"}
Oct 30 21:47:30.928397 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.928071    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f77674798-w874p" event={"ID":"9f278f63-40f1-40eb-8d8d-04f297071c1f","Type":"ContainerDied","Data":"c9b7b5da181893899130858b8aa833f281ba3671e698089aad5e3c9328bb9fed"}
Oct 30 21:47:30.928397 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.928090    5503 scope.go:122] "RemoveContainer" containerID="c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc"
Oct 30 21:47:30.938946 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.938925    5503 scope.go:122] "RemoveContainer" containerID="c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc"
Oct 30 21:47:30.939206 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:30.939182    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc\": container with ID starting with c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc not found: ID does not exist" containerID="c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc"
Oct 30 21:47:30.939270 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:30.939211    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc"} err="failed to get container status \"c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc\": rpc error: code = NotFound desc = could not find container \"c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc\": container with ID starting with c1362682596b91422f7d4aed7aabca4af2bc56a119d3e3afb7fd20770c4778bc not found: ID does not exist"
Oct 30 21:47:31.082209 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:31.082176    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f77674798-w874p"]
Oct 30 21:47:31.129031 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:31.129006    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f77674798-w874p"]
Oct 30 21:47:31.766774 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:31.766734    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvxg2"
Oct 30 21:47:31.786143 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:31.786116    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f278f63-40f1-40eb-8d8d-04f297071c1f" path="/var/lib/kubelet/pods/9f278f63-40f1-40eb-8d8d-04f297071c1f/volumes"
Oct 30 21:47:36.890464 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:36.890432    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tv4l4"
Oct 30 21:47:37.566999 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:37.566947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:37.726649 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:37.726609    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:45.757218 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:45.757170    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:46.020609 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:46.020524    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:48.065126 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.065099    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:47:48.065546 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.065427    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="alertmanager" containerID="cri-o://2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9" gracePeriod=120
Oct 30 21:47:48.065546 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.065479    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-metric" containerID="cri-o://96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062" gracePeriod=120
Oct 30 21:47:48.065546 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.065508    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="prom-label-proxy" containerID="cri-o://d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3" gracePeriod=120
Oct 30 21:47:48.066233 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.065566    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="config-reloader" containerID="cri-o://f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff" gracePeriod=120
Oct 30 21:47:48.066233 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.065570    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy" containerID="cri-o://c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342" gracePeriod=120
Oct 30 21:47:48.066233 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.066193    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-web" containerID="cri-o://db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb" gracePeriod=120
Oct 30 21:47:48.990751 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990718    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3" exitCode=0
Oct 30 21:47:48.990751 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990744    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342" exitCode=0
Oct 30 21:47:48.990751 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990751    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff" exitCode=0
Oct 30 21:47:48.990981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990759    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9" exitCode=0
Oct 30 21:47:48.990981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990788    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"}
Oct 30 21:47:48.990981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990877    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"}
Oct 30 21:47:48.990981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990906    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"}
Oct 30 21:47:48.990981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:48.990948    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"}
Oct 30 21:47:49.306112 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.306090    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:49.414157 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414128    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414157 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414159    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414189    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414218    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414246    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414277    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414309    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414354 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414337    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414699 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414678    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414767 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414736    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414821 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414767    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414821 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414814    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.414934 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.414861    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\") pod \"abc41423-0f20-491b-b9ee-fff3583ca471\" (UID: \"abc41423-0f20-491b-b9ee-fff3583ca471\") "
Oct 30 21:47:49.417461 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.417313    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:49.417461 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.417373    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:47:49.417461 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.417438    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:49.417698 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.417670    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:49.417932 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.417905    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:49.418049 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.418028    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:49.418118 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.418044    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "kube-api-access-jbrcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:49.418196 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.418170    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:49.418447 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.418425    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:49.419110 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.419092    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:49.419512 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.419496    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:49.422071 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.422034    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:49.427590 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.427567    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config" pod "abc41423-0f20-491b-b9ee-fff3583ca471" (UID: "abc41423-0f20-491b-b9ee-fff3583ca471"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:47:49.515519 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515477    5503 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515519 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515516    5503 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515519 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515526    5503 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-cluster-tls-config\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515519 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515535    5503 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515544    5503 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-tls-assets\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515553    5503 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515562    5503 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515572    5503 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/abc41423-0f20-491b-b9ee-fff3583ca471-metrics-client-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515580    5503 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-alertmanager-main-db\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515609    5503 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-web-config\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515618    5503 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/abc41423-0f20-491b-b9ee-fff3583ca471-secret-alertmanager-main-tls\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515627    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbrcq\" (UniqueName: \"kubernetes.io/projected/abc41423-0f20-491b-b9ee-fff3583ca471-kube-api-access-jbrcq\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.515793 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.515651    5503 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/abc41423-0f20-491b-b9ee-fff3583ca471-config-out\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:47:49.625679 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.625592    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:49.997147 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.997116    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062" exitCode=0
Oct 30 21:47:49.997147 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.997140    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc41423-0f20-491b-b9ee-fff3583ca471" containerID="db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb" exitCode=0
Oct 30 21:47:49.997357 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.997158    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"}
Oct 30 21:47:49.997357 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.997194    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:49.997357 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.997212    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"}
Oct 30 21:47:49.997357 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.997240    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"abc41423-0f20-491b-b9ee-fff3583ca471","Type":"ContainerDied","Data":"5829e5e450598e99e4b7aed0d2a998959f4612b4589323e0738603ff76ed62ec"}
Oct 30 21:47:49.997357 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:49.997270    5503 scope.go:122] "RemoveContainer" containerID="d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"
Oct 30 21:47:50.004256 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.004232    5503 scope.go:122] "RemoveContainer" containerID="96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"
Oct 30 21:47:50.010563 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.010547    5503 scope.go:122] "RemoveContainer" containerID="c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"
Oct 30 21:47:50.016824 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.016809    5503 scope.go:122] "RemoveContainer" containerID="db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"
Oct 30 21:47:50.022783 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.022761    5503 scope.go:122] "RemoveContainer" containerID="f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"
Oct 30 21:47:50.029176 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.029153    5503 scope.go:122] "RemoveContainer" containerID="2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"
Oct 30 21:47:50.035470 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.035453    5503 scope.go:122] "RemoveContainer" containerID="24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e"
Oct 30 21:47:50.041590 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.041574    5503 scope.go:122] "RemoveContainer" containerID="d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"
Oct 30 21:47:50.041871 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:50.041853    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3\": container with ID starting with d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3 not found: ID does not exist" containerID="d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"
Oct 30 21:47:50.041949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.041883    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"} err="failed to get container status \"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3\": rpc error: code = NotFound desc = could not find container \"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3\": container with ID starting with d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3 not found: ID does not exist"
Oct 30 21:47:50.041949 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.041907    5503 scope.go:122] "RemoveContainer" containerID="96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"
Oct 30 21:47:50.042158 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:50.042140    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062\": container with ID starting with 96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062 not found: ID does not exist" containerID="96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"
Oct 30 21:47:50.042194 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042165    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"} err="failed to get container status \"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062\": rpc error: code = NotFound desc = could not find container \"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062\": container with ID starting with 96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062 not found: ID does not exist"
Oct 30 21:47:50.042194 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042180    5503 scope.go:122] "RemoveContainer" containerID="c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"
Oct 30 21:47:50.042404 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:50.042386    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342\": container with ID starting with c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342 not found: ID does not exist" containerID="c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"
Oct 30 21:47:50.042442 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042409    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"} err="failed to get container status \"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342\": rpc error: code = NotFound desc = could not find container \"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342\": container with ID starting with c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342 not found: ID does not exist"
Oct 30 21:47:50.042442 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042423    5503 scope.go:122] "RemoveContainer" containerID="db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"
Oct 30 21:47:50.042711 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:50.042692    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb\": container with ID starting with db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb not found: ID does not exist" containerID="db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"
Oct 30 21:47:50.042780 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042714    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"} err="failed to get container status \"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb\": rpc error: code = NotFound desc = could not find container \"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb\": container with ID starting with db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb not found: ID does not exist"
Oct 30 21:47:50.042780 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042727    5503 scope.go:122] "RemoveContainer" containerID="f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"
Oct 30 21:47:50.042972 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:50.042954    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff\": container with ID starting with f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff not found: ID does not exist" containerID="f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"
Oct 30 21:47:50.043016 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042974    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"} err="failed to get container status \"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff\": rpc error: code = NotFound desc = could not find container \"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff\": container with ID starting with f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff not found: ID does not exist"
Oct 30 21:47:50.043016 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.042987    5503 scope.go:122] "RemoveContainer" containerID="2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"
Oct 30 21:47:50.043213 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:50.043199    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9\": container with ID starting with 2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9 not found: ID does not exist" containerID="2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"
Oct 30 21:47:50.043253 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.043219    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"} err="failed to get container status \"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9\": rpc error: code = NotFound desc = could not find container \"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9\": container with ID starting with 2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9 not found: ID does not exist"
Oct 30 21:47:50.043253 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.043233    5503 scope.go:122] "RemoveContainer" containerID="24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e"
Oct 30 21:47:50.043447 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:50.043430    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e\": container with ID starting with 24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e not found: ID does not exist" containerID="24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e"
Oct 30 21:47:50.043482 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.043453    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e"} err="failed to get container status \"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e\": rpc error: code = NotFound desc = could not find container \"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e\": container with ID starting with 24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e not found: ID does not exist"
Oct 30 21:47:50.043482 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.043471    5503 scope.go:122] "RemoveContainer" containerID="d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"
Oct 30 21:47:50.044070 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.043986    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3"} err="failed to get container status \"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3\": rpc error: code = NotFound desc = could not find container \"d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3\": container with ID starting with d23ef1440316abbf7ce5047cca2e54224996280ccf8eac3c1196208873be69b3 not found: ID does not exist"
Oct 30 21:47:50.044070 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.044008    5503 scope.go:122] "RemoveContainer" containerID="96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"
Oct 30 21:47:50.045869 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.045848    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062"} err="failed to get container status \"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062\": rpc error: code = NotFound desc = could not find container \"96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062\": container with ID starting with 96fc2ae85a7e6eb57c2d61db60ff55a36d625eb9692926427b49db9ec0545062 not found: ID does not exist"
Oct 30 21:47:50.045869 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.045870    5503 scope.go:122] "RemoveContainer" containerID="c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"
Oct 30 21:47:50.046730 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.046701    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342"} err="failed to get container status \"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342\": rpc error: code = NotFound desc = could not find container \"c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342\": container with ID starting with c7f7dd7b4d8da8d86ee039934ecf633bcf9fd3eaf4c95de491858c8169c94342 not found: ID does not exist"
Oct 30 21:47:50.046730 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.046719    5503 scope.go:122] "RemoveContainer" containerID="db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"
Oct 30 21:47:50.046970 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.046954    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb"} err="failed to get container status \"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb\": rpc error: code = NotFound desc = could not find container \"db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb\": container with ID starting with db18b75921eababdfff2a2f527f1b4ad1a6be6438f61d1f0dd8c7a88dc3687eb not found: ID does not exist"
Oct 30 21:47:50.047029 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.046975    5503 scope.go:122] "RemoveContainer" containerID="f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"
Oct 30 21:47:50.047239 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.047221    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff"} err="failed to get container status \"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff\": rpc error: code = NotFound desc = could not find container \"f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff\": container with ID starting with f0a1082a7d6cd0a8ddc56e4a5430d55fdb17265df0c5aff7f3d682ff3abe73ff not found: ID does not exist"
Oct 30 21:47:50.047290 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.047240    5503 scope.go:122] "RemoveContainer" containerID="2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"
Oct 30 21:47:50.047499 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.047474    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9"} err="failed to get container status \"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9\": rpc error: code = NotFound desc = could not find container \"2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9\": container with ID starting with 2d0568997635c06d2d7d95efe17ef27cc6bf29c1de1c5ad915585d4f171cafa9 not found: ID does not exist"
Oct 30 21:47:50.047561 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.047500    5503 scope.go:122] "RemoveContainer" containerID="24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e"
Oct 30 21:47:50.047843 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.047824    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e"} err="failed to get container status \"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e\": rpc error: code = NotFound desc = could not find container \"24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e\": container with ID starting with 24f4bd000f88fb45316f98a40a623c5196e7a62c7514fae821ed98cf0362ea1e not found: ID does not exist"
Oct 30 21:47:50.157385 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.157362    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:47:50.204735 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.204702    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:47:50.204964 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.204949    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:47:50.205202 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205190    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="prom-label-proxy"
Oct 30 21:47:50.205236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205203    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="prom-label-proxy"
Oct 30 21:47:50.205236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205211    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-web"
Oct 30 21:47:50.205236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205216    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-web"
Oct 30 21:47:50.205236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205226    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-metric"
Oct 30 21:47:50.205236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205231    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-metric"
Oct 30 21:47:50.205236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205238    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="config-reloader"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205243    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="config-reloader"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205250    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="alertmanager"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205254    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="alertmanager"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205262    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="init-config-reloader"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205267    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="init-config-reloader"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205274    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205278    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205287    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9f278f63-40f1-40eb-8d8d-04f297071c1f" containerName="console"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205293    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f278f63-40f1-40eb-8d8d-04f297071c1f" containerName="console"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205339    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="prom-label-proxy"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205346    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-web"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205352    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy-metric"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205358    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="config-reloader"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205364    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="alertmanager"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205369    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="init-config-reloader"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205374    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" containerName="kube-rbac-proxy"
Oct 30 21:47:50.205429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.205380    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f278f63-40f1-40eb-8d8d-04f297071c1f" containerName="console"
Oct 30 21:47:50.208032 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.208015    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221186 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221162    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221282 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221195    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221282 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221213    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6747dd9f-877b-41f4-b482-b2d4500a6cff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221282 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221228    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt84v\" (UniqueName: \"kubernetes.io/projected/6747dd9f-877b-41f4-b482-b2d4500a6cff-kube-api-access-kt84v\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221390 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221343    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-web-config\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221390 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221381    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6747dd9f-877b-41f4-b482-b2d4500a6cff-config-out\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221450 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221403    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221483 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221457    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-config-volume\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221524 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221512    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6747dd9f-877b-41f4-b482-b2d4500a6cff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221556 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221532    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221556 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221551    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221675 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221652    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6747dd9f-877b-41f4-b482-b2d4500a6cff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.221726 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.221710    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6747dd9f-877b-41f4-b482-b2d4500a6cff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.258744 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258630    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:47:50.258744 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258692    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:47:50.258744 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258707    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:47:50.260981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258752    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:47:50.260981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258758    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:47:50.260981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258792    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-r29xw\""
Oct 30 21:47:50.260981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258795    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:47:50.260981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258849    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:47:50.260981 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.258886    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:47:50.305481 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.305460    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:47:50.322393 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322364    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6747dd9f-877b-41f4-b482-b2d4500a6cff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322401    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322427    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322471    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6747dd9f-877b-41f4-b482-b2d4500a6cff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322504    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6747dd9f-877b-41f4-b482-b2d4500a6cff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322554    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322583    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322610    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6747dd9f-877b-41f4-b482-b2d4500a6cff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322678    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt84v\" (UniqueName: \"kubernetes.io/projected/6747dd9f-877b-41f4-b482-b2d4500a6cff-kube-api-access-kt84v\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322712    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-web-config\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322747    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6747dd9f-877b-41f4-b482-b2d4500a6cff-config-out\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322775    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322801    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-config-volume\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.322854    5503 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:47:50.324040 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.324004    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6747dd9f-877b-41f4-b482-b2d4500a6cff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.324142 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.324065    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6747dd9f-877b-41f4-b482-b2d4500a6cff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.324306 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.324280    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6747dd9f-877b-41f4-b482-b2d4500a6cff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.325967 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.325941    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6747dd9f-877b-41f4-b482-b2d4500a6cff-config-out\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.326478 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.326410    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6747dd9f-877b-41f4-b482-b2d4500a6cff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.326478 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.326411    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.326839 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.326723    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-config-volume\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.327352 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.327301    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.327901 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.327881    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-web-config\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.328124 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.328101    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.328198 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.328180    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.328625 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.328602    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6747dd9f-877b-41f4-b482-b2d4500a6cff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.386112 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.386082    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt84v\" (UniqueName: \"kubernetes.io/projected/6747dd9f-877b-41f4-b482-b2d4500a6cff-kube-api-access-kt84v\") pod \"alertmanager-main-0\" (UID: \"6747dd9f-877b-41f4-b482-b2d4500a6cff\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.432651 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.432608    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:47:50.523236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.523148    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:47:50.943550 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:50.943518    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:47:50.946419 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:50.946387    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6747dd9f_877b_41f4_b482_b2d4500a6cff.slice/crio-b27c89627120ea47c8778f14e251ab179a50ebf9c57f1b4f55a435aa33e1be06 WatchSource:0}: Error finding container b27c89627120ea47c8778f14e251ab179a50ebf9c57f1b4f55a435aa33e1be06: Status 404 returned error can't find the container with id b27c89627120ea47c8778f14e251ab179a50ebf9c57f1b4f55a435aa33e1be06
Oct 30 21:47:51.003118 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.003090    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"e31b08beb7615bab048a383308dbab05d66a9e689c95388c6af6e6a49397384f"}
Oct 30 21:47:51.003237 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.003172    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"b27c89627120ea47c8778f14e251ab179a50ebf9c57f1b4f55a435aa33e1be06"}
Oct 30 21:47:51.696491 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.696456    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7458fbb99c-q7l58"]
Oct 30 21:47:51.701502 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.701478    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734190 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734160    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-serving-certs-ca-bundle\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734324 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734198    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74tkp\" (UniqueName: \"kubernetes.io/projected/0b8454ef-998f-4574-ae4e-ab0c094a7562-kube-api-access-74tkp\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734324 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734232    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-telemeter-client-tls\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734400 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734340    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-secret-telemeter-client\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734400 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734373    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-metrics-client-ca\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734471 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734447    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734513 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734489    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-federate-client-tls\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.734513 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.734506    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.753299 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.753275    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 21:47:51.753570 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.753550    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 21:47:51.753647 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.753556    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 21:47:51.786740 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.786713    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc41423-0f20-491b-b9ee-fff3583ca471" path="/var/lib/kubelet/pods/abc41423-0f20-491b-b9ee-fff3583ca471/volumes"
Oct 30 21:47:51.800336 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.800317    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 21:47:51.800336 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.800333    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 21:47:51.800857 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.800840    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wmw89\""
Oct 30 21:47:51.835369 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835345    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-federate-client-tls\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.835474 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835372    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.835474 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835392    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-serving-certs-ca-bundle\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.835474 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835411    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tkp\" (UniqueName: \"kubernetes.io/projected/0b8454ef-998f-4574-ae4e-ab0c094a7562-kube-api-access-74tkp\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.835474 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835434    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-telemeter-client-tls\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.835474 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835452    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-secret-telemeter-client\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.835749 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835512    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-metrics-client-ca\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.835749 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.835580    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.836203 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.836173    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-serving-certs-ca-bundle\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.836331 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.836303    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-metrics-client-ca\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.838050 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.838025    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-telemeter-client-tls\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.838133 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.838111    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.838174 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.838138    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-federate-client-tls\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.838174 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.838151    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b8454ef-998f-4574-ae4e-ab0c094a7562-secret-telemeter-client\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.848825 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.848803    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 21:47:51.849383 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.849364    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7458fbb99c-q7l58"]
Oct 30 21:47:51.856734 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.856710    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b8454ef-998f-4574-ae4e-ab0c094a7562-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:51.894446 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:51.894415    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74tkp\" (UniqueName: \"kubernetes.io/projected/0b8454ef-998f-4574-ae4e-ab0c094a7562-kube-api-access-74tkp\") pod \"telemeter-client-7458fbb99c-q7l58\" (UID: \"0b8454ef-998f-4574-ae4e-ab0c094a7562\") " pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:52.009606 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:52.009527    5503 generic.go:356] "Generic (PLEG): container finished" podID="6747dd9f-877b-41f4-b482-b2d4500a6cff" containerID="e31b08beb7615bab048a383308dbab05d66a9e689c95388c6af6e6a49397384f" exitCode=0
Oct 30 21:47:52.009606 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:52.009586    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerDied","Data":"e31b08beb7615bab048a383308dbab05d66a9e689c95388c6af6e6a49397384f"}
Oct 30 21:47:52.012902 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:52.012881    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58"
Oct 30 21:47:52.436539 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:52.436505    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7458fbb99c-q7l58"]
Oct 30 21:47:52.440294 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:52.440270    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8454ef_998f_4574_ae4e_ab0c094a7562.slice/crio-76768a1a0bfe9e0ffb34de05c708bf3db2be452fc2096ef81370e4b36031bffc WatchSource:0}: Error finding container 76768a1a0bfe9e0ffb34de05c708bf3db2be452fc2096ef81370e4b36031bffc: Status 404 returned error can't find the container with id 76768a1a0bfe9e0ffb34de05c708bf3db2be452fc2096ef81370e4b36031bffc
Oct 30 21:47:52.442933 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:52.442914    5503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:47:53.017519 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.017483    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58" event={"ID":"0b8454ef-998f-4574-ae4e-ab0c094a7562","Type":"ContainerStarted","Data":"76768a1a0bfe9e0ffb34de05c708bf3db2be452fc2096ef81370e4b36031bffc"}
Oct 30 21:47:53.021652 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.021616    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"b5bdb61a238fa49fd18fbe6c35bf57c08f0b1bfd0e6961f03aa1bdf5a8128173"}
Oct 30 21:47:53.021787 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.021698    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"589170c8a12c4a184d131f235890ab9a1a798e899d96907eb2d8fe652ec32e93"}
Oct 30 21:47:53.021787 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.021738    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"32c2a996758dde36e495eef77921f36599ffa58279158b33442c7cfc84731935"}
Oct 30 21:47:53.021787 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.021774    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"bdac535cd5d23cb0c5fb17a96ddd1dff1529f37446e83e73b874c6fe322c643b"}
Oct 30 21:47:53.021898 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.021816    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"7c402ef7328e7649e512dfed1f99b55f69344b1cfa973dce0304dc23cc730878"}
Oct 30 21:47:53.021898 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.021850    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6747dd9f-877b-41f4-b482-b2d4500a6cff","Type":"ContainerStarted","Data":"6c95e41d0a345791260e714f8c7942fbd3f1068e53e2c8d8c1f37b571116b0fa"}
Oct 30 21:47:53.146506 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:53.146455    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.130655007 podStartE2EDuration="3.146442001s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.786994ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:53.145520956 +0000 UTC m=+49.480097742" watchObservedRunningTime="2026-10-30 21:47:53.146442001 +0000 UTC m=+49.481018770"
Oct 30 21:47:55.031617 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.031582    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58" event={"ID":"0b8454ef-998f-4574-ae4e-ab0c094a7562","Type":"ContainerStarted","Data":"fe2ae8d9a939c4c072deadf9f60371afde340fee4edad3fd1022045aef789e55"}
Oct 30 21:47:55.032037 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.031673    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58" event={"ID":"0b8454ef-998f-4574-ae4e-ab0c094a7562","Type":"ContainerStarted","Data":"95f285929463cadc604e5fd0918ac30fe2b2b12aea26581c48d8085ac0cf6fb2"}
Oct 30 21:47:55.032037 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.031695    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58" event={"ID":"0b8454ef-998f-4574-ae4e-ab0c094a7562","Type":"ContainerStarted","Data":"ba0852307cde845436d6d38a9ab60c4b96462d8c2d837f4fb268cbf9a94d51d8"}
Oct 30 21:47:55.243445 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.243391    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7458fbb99c-q7l58" podStartSLOduration=2.05675064 podStartE2EDuration="4.243378457s" totalImagesPullingTime="2.186627817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:55.196355845 +0000 UTC m=+51.530932652" watchObservedRunningTime="2026-10-30 21:47:55.243378457 +0000 UTC m=+51.577955225"
Oct 30 21:47:55.640824 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.640794    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b867c87d8-7hx2f"]
Oct 30 21:47:55.643153 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.643131    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667517 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667495    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-oauth-config\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667616 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667532    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b503b18a-2304-4fce-8308-af84d14f94e7-tmp\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667616 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667552    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-serving-cert\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667726 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667669    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-service-ca\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667726 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667698    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-trusted-ca-bundle\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667726 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667719    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-console-config\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667838 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667737    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt4vv\" (UniqueName: \"kubernetes.io/projected/b503b18a-2304-4fce-8308-af84d14f94e7-kube-api-access-pt4vv\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.667838 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.667807    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-oauth-serving-cert\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.692741 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.692722    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:47:55.692878 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.692863    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:47:55.692878 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.692875    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:47:55.693014 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.692952    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:47:55.693014 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.693010    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:47:55.693109 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.693055    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:47:55.693225 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.693212    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:47:55.693275 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.693254    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cpxgn\""
Oct 30 21:47:55.698028 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.698006    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:55.768933 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.768909    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-service-ca\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769048 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.768938    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-trusted-ca-bundle\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769048 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.768957    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-console-config\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769048 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.768975    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt4vv\" (UniqueName: \"kubernetes.io/projected/b503b18a-2304-4fce-8308-af84d14f94e7-kube-api-access-pt4vv\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769048 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.768991    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-oauth-serving-cert\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.769043    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-oauth-config\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.769110    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b503b18a-2304-4fce-8308-af84d14f94e7-tmp\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.769146    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-serving-cert\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769557 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.769535    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b503b18a-2304-4fce-8308-af84d14f94e7-tmp\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769711 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.769689    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-service-ca\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769789 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.769752    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-oauth-serving-cert\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.769789 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.769761    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-console-config\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.771430 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.771412    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-oauth-config\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.771491 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.771475    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-serving-cert\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.787071 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.787045    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:47:55.787651 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.787613    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b867c87d8-7hx2f"]
Oct 30 21:47:55.790582 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.790562    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-trusted-ca-bundle\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.833222 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.833201    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt4vv\" (UniqueName: \"kubernetes.io/projected/b503b18a-2304-4fce-8308-af84d14f94e7-kube-api-access-pt4vv\") pod \"console-7b867c87d8-7hx2f\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") " pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:55.954464 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.954436    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:55.957370 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:55.957345    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:47:56.114902 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:56.114861    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:56.336513 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:56.336436    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b867c87d8-7hx2f"]
Oct 30 21:47:56.339558 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:47:56.339532    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb503b18a_2304_4fce_8308_af84d14f94e7.slice/crio-a6466687287a3de3b9ce30fa5f52fc03bc5836a7d2511511184566f16606e3ac WatchSource:0}: Error finding container a6466687287a3de3b9ce30fa5f52fc03bc5836a7d2511511184566f16606e3ac: Status 404 returned error can't find the container with id a6466687287a3de3b9ce30fa5f52fc03bc5836a7d2511511184566f16606e3ac
Oct 30 21:47:56.345515 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:56.345489    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:56.498921 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:56.498886    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:47:57.053775 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:57.053739    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b867c87d8-7hx2f" event={"ID":"b503b18a-2304-4fce-8308-af84d14f94e7","Type":"ContainerStarted","Data":"5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8"}
Oct 30 21:47:57.053946 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:57.053788    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b867c87d8-7hx2f" event={"ID":"b503b18a-2304-4fce-8308-af84d14f94e7","Type":"ContainerStarted","Data":"a6466687287a3de3b9ce30fa5f52fc03bc5836a7d2511511184566f16606e3ac"}
Oct 30 21:47:57.169982 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:47:57.169929    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b867c87d8-7hx2f" podStartSLOduration=2.169914408 podStartE2EDuration="2.169914408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:57.169428581 +0000 UTC m=+53.504005350" watchObservedRunningTime="2026-10-30 21:47:57.169914408 +0000 UTC m=+53.504491177"
Oct 30 21:47:58.102391 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:47:58.102361    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:48:05.957900 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:05.957872    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:48:05.958364 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:05.957921    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:48:05.962664 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:05.962631    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:48:06.088247 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:06.088223    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:48:09.629614 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:09.629553    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:13.441456 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:13.441426    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:13.598014 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:13.597985    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:14.161026 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.160995    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-578bc7c998-rxh9z"]
Oct 30 21:48:14.162804 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.162787    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230361 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230330    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-service-ca\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230495 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230363    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-trusted-ca-bundle\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230495 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230391    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-config\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230569 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230511    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-oauth-serving-cert\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230569 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230545    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-oauth-config\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230664 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230569    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-serving-cert\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230664 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230648    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkv5\" (UniqueName: \"kubernetes.io/projected/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-kube-api-access-9zkv5\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.230748 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.230733    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-tmp\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.260215 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.260185    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-578bc7c998-rxh9z"]
Oct 30 21:48:14.331512 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331484    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-oauth-config\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.331688 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331519    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-serving-cert\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.331688 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331536    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkv5\" (UniqueName: \"kubernetes.io/projected/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-kube-api-access-9zkv5\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.331688 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331562    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-tmp\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.331688 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331601    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-service-ca\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.331688 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331619    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-trusted-ca-bundle\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.331688 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331653    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-config\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.332002 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.331694    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-oauth-serving-cert\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.332168 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.332145    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-tmp\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.332570 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.332546    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-service-ca\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.332681 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.332665    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-oauth-serving-cert\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.332728 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.332677    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-trusted-ca-bundle\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.332815 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.332798    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-config\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.334534 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.334513    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-oauth-config\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.334730 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.334711    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-console-serving-cert\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.390483 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.390454    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkv5\" (UniqueName: \"kubernetes.io/projected/144a8617-7262-4aa3-9f36-c7bcc4a5c09a-kube-api-access-9zkv5\") pod \"console-578bc7c998-rxh9z\" (UID: \"144a8617-7262-4aa3-9f36-c7bcc4a5c09a\") " pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.459341 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.459282    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:14.472385 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.472366    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:14.634600 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.634567    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:14.838529 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.838455    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-578bc7c998-rxh9z"]
Oct 30 21:48:14.841554 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:48:14.841525    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144a8617_7262_4aa3_9f36_c7bcc4a5c09a.slice/crio-4265a3e12d0bc5953246cf167f2e193cb2820d877030b960cbb96f069cf9171a WatchSource:0}: Error finding container 4265a3e12d0bc5953246cf167f2e193cb2820d877030b960cbb96f069cf9171a: Status 404 returned error can't find the container with id 4265a3e12d0bc5953246cf167f2e193cb2820d877030b960cbb96f069cf9171a
Oct 30 21:48:14.862906 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:14.862881    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:15.016239 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:15.016211    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:15.120748 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:15.120680    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-578bc7c998-rxh9z" event={"ID":"144a8617-7262-4aa3-9f36-c7bcc4a5c09a","Type":"ContainerStarted","Data":"ade4089befbcdf313dc7ec94f294d1f1ced21aea370aca38200a90cdce8574fe"}
Oct 30 21:48:15.120748 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:15.120725    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-578bc7c998-rxh9z" event={"ID":"144a8617-7262-4aa3-9f36-c7bcc4a5c09a","Type":"ContainerStarted","Data":"4265a3e12d0bc5953246cf167f2e193cb2820d877030b960cbb96f069cf9171a"}
Oct 30 21:48:15.234345 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:15.234293    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-578bc7c998-rxh9z" podStartSLOduration=1.234279204 podStartE2EDuration="1.234279204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:15.233718267 +0000 UTC m=+71.568295037" watchObservedRunningTime="2026-10-30 21:48:15.234279204 +0000 UTC m=+71.568855967"
Oct 30 21:48:24.473181 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:24.473146    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:24.473664 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:24.473330    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:24.478119 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:24.478102    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:25.158230 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:25.158205    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-578bc7c998-rxh9z"
Oct 30 21:48:25.318479 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:25.318442    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b867c87d8-7hx2f"]
Oct 30 21:48:26.077271 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:26.077238    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:26.239480 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:26.239441    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:28.096597 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:48:28.096560    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:48:42.869107 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:42.869022    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:43.078564 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:43.078523    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:48:50.337443 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.337405    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b867c87d8-7hx2f" podUID="b503b18a-2304-4fce-8308-af84d14f94e7" containerName="console" containerID="cri-o://5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8" gracePeriod=15
Oct 30 21:48:50.582877 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.582852    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:48:50.600610 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600553    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-oauth-serving-cert\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600610 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600599    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-trusted-ca-bundle\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600768 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600674    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-console-config\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-console-config\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600768 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600695    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-oauth-config\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600768 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600714    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-serving-cert\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600768 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600734    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b503b18a-2304-4fce-8308-af84d14f94e7-kube-api-access-pt4vv\" (UniqueName: \"kubernetes.io/projected/b503b18a-2304-4fce-8308-af84d14f94e7-kube-api-access-pt4vv\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600991 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600791    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-service-ca\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-service-ca\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600991 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600827    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b503b18a-2304-4fce-8308-af84d14f94e7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b503b18a-2304-4fce-8308-af84d14f94e7-tmp\") pod \"b503b18a-2304-4fce-8308-af84d14f94e7\" (UID: \"b503b18a-2304-4fce-8308-af84d14f94e7\") "
Oct 30 21:48:50.600991 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.600945    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-oauth-serving-cert" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:50.601133 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.601045    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-trusted-ca-bundle" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:50.601133 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.601113    5503 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-oauth-serving-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.601256 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.601235    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b503b18a-2304-4fce-8308-af84d14f94e7-tmp" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:50.601256 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.601152    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-console-config" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:50.601606 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.601575    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-service-ca" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:50.603213 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.603185    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-oauth-config" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:50.603310 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.603270    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-serving-cert" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:50.603372 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.603346    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b503b18a-2304-4fce-8308-af84d14f94e7-kube-api-access-pt4vv" pod "b503b18a-2304-4fce-8308-af84d14f94e7" (UID: "b503b18a-2304-4fce-8308-af84d14f94e7"). InnerVolumeSpecName "kube-api-access-pt4vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:50.701532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.701497    5503 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-trusted-ca-bundle\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.701532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.701529    5503 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-console-config\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.701532 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.701540    5503 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-oauth-config\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.701765 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.701549    5503 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b503b18a-2304-4fce-8308-af84d14f94e7-console-serving-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.701765 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.701558    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt4vv\" (UniqueName: \"kubernetes.io/projected/b503b18a-2304-4fce-8308-af84d14f94e7-kube-api-access-pt4vv\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.701765 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.701568    5503 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b503b18a-2304-4fce-8308-af84d14f94e7-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:50.701765 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:50.701577    5503 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b503b18a-2304-4fce-8308-af84d14f94e7-tmp\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:51.246947 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.246910    5503 generic.go:356] "Generic (PLEG): container finished" podID="b503b18a-2304-4fce-8308-af84d14f94e7" containerID="5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8" exitCode=0
Oct 30 21:48:51.247108 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.246982    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b867c87d8-7hx2f"
Oct 30 21:48:51.247108 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.246996    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b867c87d8-7hx2f" event={"ID":"b503b18a-2304-4fce-8308-af84d14f94e7","Type":"ContainerDied","Data":"5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8"}
Oct 30 21:48:51.247108 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.247067    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b867c87d8-7hx2f" event={"ID":"b503b18a-2304-4fce-8308-af84d14f94e7","Type":"ContainerDied","Data":"a6466687287a3de3b9ce30fa5f52fc03bc5836a7d2511511184566f16606e3ac"}
Oct 30 21:48:51.247108 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.247087    5503 scope.go:122] "RemoveContainer" containerID="5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8"
Oct 30 21:48:51.255179 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.255161    5503 scope.go:122] "RemoveContainer" containerID="5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8"
Oct 30 21:48:51.255446 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:48:51.255426    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8\": container with ID starting with 5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8 not found: ID does not exist" containerID="5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8"
Oct 30 21:48:51.255503 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.255454    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8"} err="failed to get container status \"5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8\": rpc error: code = NotFound desc = could not find container \"5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8\": container with ID starting with 5bf8027ee4c3b52884290046e7333740f8d603118df7d96d002578af8b478da8 not found: ID does not exist"
Oct 30 21:48:51.404259 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.404227    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b867c87d8-7hx2f"]
Oct 30 21:48:51.451011 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.450985    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b867c87d8-7hx2f"]
Oct 30 21:48:51.786252 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:48:51.786216    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b503b18a-2304-4fce-8308-af84d14f94e7" path="/var/lib/kubelet/pods/b503b18a-2304-4fce-8308-af84d14f94e7/volumes"
Oct 30 21:48:58.098140 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:48:58.098103    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:49:09.962835 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:49:09.962801    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:10.573947 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:49:10.573913    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:10.763710 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:49:10.763677    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:28.097478 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:49:28.097435    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:49:40.722093 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:49:40.722030    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:40.937128 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:49:40.937088    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:41.310455 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:49:41.310414    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:49:58.098040 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:49:58.098002    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:50:10.288605 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:10.288570    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:28.097047 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:50:28.096966    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:50:58.097869 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:50:58.097827    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:50:58.155498 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:58.155450    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:59.157905 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.157871    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:59.342546 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.342513    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"]
Oct 30 21:50:59.342804 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.342791    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b503b18a-2304-4fce-8308-af84d14f94e7" containerName="console"
Oct 30 21:50:59.342852 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.342805    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b503b18a-2304-4fce-8308-af84d14f94e7" containerName="console"
Oct 30 21:50:59.343025 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.342876    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b503b18a-2304-4fce-8308-af84d14f94e7" containerName="console"
Oct 30 21:50:59.344730 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.344706    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.364970 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.364948    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb84n\" (UniqueName: \"kubernetes.io/projected/0d4dd2ee-cb30-4999-b408-d570ece2f17f-kube-api-access-hb84n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.365096 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.364982    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.365167 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.365091    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.444475 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.444407    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:50:59.444475 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.444407    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:50:59.445107 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.445091    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"]
Oct 30 21:50:59.447162 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.447148    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.451111 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.451081    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"]
Oct 30 21:50:59.466552 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.466525    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.466654 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.466570    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb84n\" (UniqueName: \"kubernetes.io/projected/0d4dd2ee-cb30-4999-b408-d570ece2f17f-kube-api-access-hb84n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.466709 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.466655    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.466746 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.466700    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbvg\" (UniqueName: \"kubernetes.io/projected/7cad32b7-5c7d-4ae9-a014-4d83c520c194-kube-api-access-vbbvg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.466797 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.466743    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.466797 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.466768    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.467099 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.467077    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.467174 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.467113    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.467969 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.467950    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ctvtr\""
Oct 30 21:50:59.499154 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.499121    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"]
Oct 30 21:50:59.524522 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.524495    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb84n\" (UniqueName: \"kubernetes.io/projected/0d4dd2ee-cb30-4999-b408-d570ece2f17f-kube-api-access-hb84n\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.549861 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.549840    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7xsr9\""
Oct 30 21:50:59.567914 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.567885    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.567996 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.567945    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.567996 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.567976    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbvg\" (UniqueName: \"kubernetes.io/projected/7cad32b7-5c7d-4ae9-a014-4d83c520c194-kube-api-access-vbbvg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.570143 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.568289    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.570143 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.568380    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.626695 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.626667    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbvg\" (UniqueName: \"kubernetes.io/projected/7cad32b7-5c7d-4ae9-a014-4d83c520c194-kube-api-access-vbbvg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.657443 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.657413    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:50:59.660753 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.660729    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"]
Oct 30 21:50:59.662796 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.662779    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.689106 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.689076    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:59.716527 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.716425    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:50:59.716527 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.716501    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:50:59.716701 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.716559    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:50:59.716701 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.716574    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:50:59.716701 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.716607    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:50:59.716701 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.716678    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:50:59.716889 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.716787    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7m88w\""
Oct 30 21:50:59.724275 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.724250    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"]
Oct 30 21:50:59.759773 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.759753    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:50:59.770260 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.770226    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.770378 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.770282    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.770378 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.770360    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d453e-7b20-423c-a96b-a3ab90553e2e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.770480 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.770409    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.770527 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.770508    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmq7\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-kube-api-access-plmq7\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.770577 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.770539    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.770577 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.770565    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.843174 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.843146    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:50:59.871854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.871833    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmq7\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-kube-api-access-plmq7\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.871958 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.871862    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.871958 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.871880    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.871958 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.871935    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.872086 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.871976    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.872086 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.872010    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d453e-7b20-423c-a96b-a3ab90553e2e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.872086 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.872055    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.872603 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.872578    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.874195 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.874177    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.874278 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.874208    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.874466 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.874450    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d453e-7b20-423c-a96b-a3ab90553e2e-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.874520 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.874500    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.930245 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.930213    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmq7\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-kube-api-access-plmq7\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.930370 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.930288    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-cxp9x\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:50:59.985417 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:50:59.985368    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:51:00.025437 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:00.025285    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"]
Oct 30 21:51:00.028157 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:51:00.028123    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4dd2ee_cb30_4999_b408_d570ece2f17f.slice/crio-93f989de10f6ca446904ab27fead4ea5bcd9055cac0dbe329f079c500ad5ab8b WatchSource:0}: Error finding container 93f989de10f6ca446904ab27fead4ea5bcd9055cac0dbe329f079c500ad5ab8b: Status 404 returned error can't find the container with id 93f989de10f6ca446904ab27fead4ea5bcd9055cac0dbe329f079c500ad5ab8b
Oct 30 21:51:00.120420 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:00.120394    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"]
Oct 30 21:51:00.123491 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:51:00.123466    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cad32b7_5c7d_4ae9_a014_4d83c520c194.slice/crio-14ab80b84bd469a5070725a9ddad917654921622ed8b3e3888755c3297ae2554 WatchSource:0}: Error finding container 14ab80b84bd469a5070725a9ddad917654921622ed8b3e3888755c3297ae2554: Status 404 returned error can't find the container with id 14ab80b84bd469a5070725a9ddad917654921622ed8b3e3888755c3297ae2554
Oct 30 21:51:00.351936 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:00.351870    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"]
Oct 30 21:51:00.354852 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:51:00.354823    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6d453e_7b20_423c_a96b_a3ab90553e2e.slice/crio-378d07026225e2f839bc3abad664ac553c4eb45e644e144edf9b8cc920880f63 WatchSource:0}: Error finding container 378d07026225e2f839bc3abad664ac553c4eb45e644e144edf9b8cc920880f63: Status 404 returned error can't find the container with id 378d07026225e2f839bc3abad664ac553c4eb45e644e144edf9b8cc920880f63
Oct 30 21:51:00.682721 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:00.682679    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c" event={"ID":"7cad32b7-5c7d-4ae9-a014-4d83c520c194","Type":"ContainerStarted","Data":"14ab80b84bd469a5070725a9ddad917654921622ed8b3e3888755c3297ae2554"}
Oct 30 21:51:00.683778 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:00.683751    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk" event={"ID":"0d4dd2ee-cb30-4999-b408-d570ece2f17f","Type":"ContainerStarted","Data":"93f989de10f6ca446904ab27fead4ea5bcd9055cac0dbe329f079c500ad5ab8b"}
Oct 30 21:51:00.684886 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:00.684864    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" event={"ID":"5e6d453e-7b20-423c-a96b-a3ab90553e2e","Type":"ContainerStarted","Data":"378d07026225e2f839bc3abad664ac553c4eb45e644e144edf9b8cc920880f63"}
Oct 30 21:51:01.016331 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:01.016249    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:05.034236 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:05.034201    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 21:51:05.034674 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:05.034265    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 21:51:05.710178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:05.709954    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" event={"ID":"5e6d453e-7b20-423c-a96b-a3ab90553e2e","Type":"ContainerStarted","Data":"89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da"}
Oct 30 21:51:05.710178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:05.710072    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:51:05.938223 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:05.937288    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" podStartSLOduration=2.260659796 podStartE2EDuration="6.93726719s" totalImagesPullingTime="4.676607394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:05.889438152 +0000 UTC m=+242.224014943" watchObservedRunningTime="2026-10-30 21:51:05.93726719 +0000 UTC m=+242.271843960"
Oct 30 21:51:06.435662 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:06.435539    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:06.602878 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:06.602844    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:06.716218 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:06.716134    5503 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cxp9x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:51:06.716218 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:06.716191    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" podUID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:51:07.721052 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:07.721019    5503 generic.go:356] "Generic (PLEG): container finished" podID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerID="6afb8c706a2f020e05f1187d6e8e4aaeeaa24c3030b451fc30d93ed608b14889" exitCode=0
Oct 30 21:51:07.721458 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:07.721105    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c" event={"ID":"7cad32b7-5c7d-4ae9-a014-4d83c520c194","Type":"ContainerDied","Data":"6afb8c706a2f020e05f1187d6e8e4aaeeaa24c3030b451fc30d93ed608b14889"}
Oct 30 21:51:07.722759 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:07.722736    5503 generic.go:356] "Generic (PLEG): container finished" podID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerID="6c1e0ffa5660c5aa69515bb38426af676d927d2c0209e10bf79a8c624b671d3c" exitCode=0
Oct 30 21:51:07.722860 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:07.722824    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk" event={"ID":"0d4dd2ee-cb30-4999-b408-d570ece2f17f","Type":"ContainerDied","Data":"6c1e0ffa5660c5aa69515bb38426af676d927d2c0209e10bf79a8c624b671d3c"}
Oct 30 21:51:07.979203 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:07.979172    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:08.332207 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:08.332173    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:08.486543 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:08.486461    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:09.715895 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:09.715858    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:51:09.730953 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:09.730920    5503 generic.go:356] "Generic (PLEG): container finished" podID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerID="473521341bcb86873c356b08cf4ca025a9f8a211f070c7d16c22c1ba9e184931" exitCode=0
Oct 30 21:51:09.731112 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:09.730969    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c" event={"ID":"7cad32b7-5c7d-4ae9-a014-4d83c520c194","Type":"ContainerDied","Data":"473521341bcb86873c356b08cf4ca025a9f8a211f070c7d16c22c1ba9e184931"}
Oct 30 21:51:10.622883 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:10.622851    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:10.735816 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:10.735781    5503 generic.go:356] "Generic (PLEG): container finished" podID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerID="a1babe1e314a6b106eb8fc2f7d36c85c8b193233217a72ce3714344485c9409f" exitCode=0
Oct 30 21:51:10.736186 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:10.735865    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk" event={"ID":"0d4dd2ee-cb30-4999-b408-d570ece2f17f","Type":"ContainerDied","Data":"a1babe1e314a6b106eb8fc2f7d36c85c8b193233217a72ce3714344485c9409f"}
Oct 30 21:51:12.165478 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:12.165443    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:17.769701 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:17.769667    5503 generic.go:356] "Generic (PLEG): container finished" podID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerID="9fc7aab95d3d22c3045f61505890d2168b11d72888769f7ed7ea6a172ad71344" exitCode=0
Oct 30 21:51:17.770191 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:17.769757    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c" event={"ID":"7cad32b7-5c7d-4ae9-a014-4d83c520c194","Type":"ContainerDied","Data":"9fc7aab95d3d22c3045f61505890d2168b11d72888769f7ed7ea6a172ad71344"}
Oct 30 21:51:18.899788 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:18.899764    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:51:19.060092 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.060020    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-bundle\") pod \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") "
Oct 30 21:51:19.060092 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.060052    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-util\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-util\") pod \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") "
Oct 30 21:51:19.060262 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.060098    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cad32b7-5c7d-4ae9-a014-4d83c520c194-kube-api-access-vbbvg\" (UniqueName: \"kubernetes.io/projected/7cad32b7-5c7d-4ae9-a014-4d83c520c194-kube-api-access-vbbvg\") pod \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\" (UID: \"7cad32b7-5c7d-4ae9-a014-4d83c520c194\") "
Oct 30 21:51:19.060614 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.060586    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-bundle" pod "7cad32b7-5c7d-4ae9-a014-4d83c520c194" (UID: "7cad32b7-5c7d-4ae9-a014-4d83c520c194"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:19.062212 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.062188    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cad32b7-5c7d-4ae9-a014-4d83c520c194-kube-api-access-vbbvg" pod "7cad32b7-5c7d-4ae9-a014-4d83c520c194" (UID: "7cad32b7-5c7d-4ae9-a014-4d83c520c194"). InnerVolumeSpecName "kube-api-access-vbbvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:19.065131 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.065097    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-util" pod "7cad32b7-5c7d-4ae9-a014-4d83c520c194" (UID: "7cad32b7-5c7d-4ae9-a014-4d83c520c194"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:19.161322 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.161293    5503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-bundle\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:19.161322 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.161321    5503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7cad32b7-5c7d-4ae9-a014-4d83c520c194-util\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:19.161512 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.161334    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbvg\" (UniqueName: \"kubernetes.io/projected/7cad32b7-5c7d-4ae9-a014-4d83c520c194-kube-api-access-vbbvg\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:19.781073 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.781038    5503 generic.go:356] "Generic (PLEG): container finished" podID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerID="45449abdc5bc6b7e3458bef59be149552bc958f874542b3b1a2bf8b0dad100b6" exitCode=0
Oct 30 21:51:19.781261 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.781129    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk" event={"ID":"0d4dd2ee-cb30-4999-b408-d570ece2f17f","Type":"ContainerDied","Data":"45449abdc5bc6b7e3458bef59be149552bc958f874542b3b1a2bf8b0dad100b6"}
Oct 30 21:51:19.783092 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.783070    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c"
Oct 30 21:51:19.786080 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.786054    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7172w6c" event={"ID":"7cad32b7-5c7d-4ae9-a014-4d83c520c194","Type":"ContainerDied","Data":"14ab80b84bd469a5070725a9ddad917654921622ed8b3e3888755c3297ae2554"}
Oct 30 21:51:19.786171 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:19.786095    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14ab80b84bd469a5070725a9ddad917654921622ed8b3e3888755c3297ae2554"
Oct 30 21:51:20.906184 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:20.906159    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:51:21.075290 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.075221    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-bundle\") pod \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") "
Oct 30 21:51:21.075290 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.075253    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-util\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-util\") pod \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") "
Oct 30 21:51:21.075469 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.075312    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d4dd2ee-cb30-4999-b408-d570ece2f17f-kube-api-access-hb84n\" (UniqueName: \"kubernetes.io/projected/0d4dd2ee-cb30-4999-b408-d570ece2f17f-kube-api-access-hb84n\") pod \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\" (UID: \"0d4dd2ee-cb30-4999-b408-d570ece2f17f\") "
Oct 30 21:51:21.076056 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.076017    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-bundle" pod "0d4dd2ee-cb30-4999-b408-d570ece2f17f" (UID: "0d4dd2ee-cb30-4999-b408-d570ece2f17f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:21.077422 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.077399    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4dd2ee-cb30-4999-b408-d570ece2f17f-kube-api-access-hb84n" pod "0d4dd2ee-cb30-4999-b408-d570ece2f17f" (UID: "0d4dd2ee-cb30-4999-b408-d570ece2f17f"). InnerVolumeSpecName "kube-api-access-hb84n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:21.082873 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.082845    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-util" pod "0d4dd2ee-cb30-4999-b408-d570ece2f17f" (UID: "0d4dd2ee-cb30-4999-b408-d570ece2f17f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:51:21.175989 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.175951    5503 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-bundle\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:21.175989 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.175981    5503 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d4dd2ee-cb30-4999-b408-d570ece2f17f-util\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:21.175989 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.175991    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb84n\" (UniqueName: \"kubernetes.io/projected/0d4dd2ee-cb30-4999-b408-d570ece2f17f-kube-api-access-hb84n\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:51:21.792205 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.792165    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk" event={"ID":"0d4dd2ee-cb30-4999-b408-d570ece2f17f","Type":"ContainerDied","Data":"93f989de10f6ca446904ab27fead4ea5bcd9055cac0dbe329f079c500ad5ab8b"}
Oct 30 21:51:21.792382 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.792215    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93f989de10f6ca446904ab27fead4ea5bcd9055cac0dbe329f079c500ad5ab8b"
Oct 30 21:51:21.792382 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:21.792177    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cjvqwk"
Oct 30 21:51:28.098218 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:51:28.098177    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:51:42.739929 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:42.739873    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:42.972199 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:42.972139    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:51.845400 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:51.845349    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:55.646629 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:55.646560    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:55.860335 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:55.860300    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:56.874470 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:51:56.874434    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:51:58.098415 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:51:58.098380    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:52:01.248961 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:01.248928    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:01.507486 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:01.507389    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:01.729368 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:01.729330    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:01.934161 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:01.934113    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:02.088386 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:02.088346    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:02.261452 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:02.261336    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:03.125536 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.125455    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:03.367471 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.367398    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:03.678747 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.678709    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"]
Oct 30 21:52:03.679091 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679074    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerName="pull"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679093    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerName="pull"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679109    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerName="util"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679115    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerName="util"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679125    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerName="extract"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679133    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerName="extract"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679157    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerName="pull"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679165    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerName="pull"
Oct 30 21:52:03.679178 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679176    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerName="util"
Oct 30 21:52:03.679564 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679183    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerName="util"
Oct 30 21:52:03.679564 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679197    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerName="extract"
Oct 30 21:52:03.679564 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679204    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerName="extract"
Oct 30 21:52:03.679564 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679277    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cad32b7-5c7d-4ae9-a014-4d83c520c194" containerName="extract"
Oct 30 21:52:03.679564 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.679293    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d4dd2ee-cb30-4999-b408-d570ece2f17f" containerName="extract"
Oct 30 21:52:03.683450 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.683429    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.790405 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.790377    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.790575 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.790436    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.790575 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.790477    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgzc\" (UniqueName: \"kubernetes.io/projected/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-kube-api-access-fjgzc\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.790736 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.790601    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.790736 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.790684    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.790837 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.790744    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.795394 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.795348    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:52:03.795528 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.795509    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:52:03.795776 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.795763    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:52:03.795824 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.795780    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:52:03.795894 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.795876    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:52:03.796051 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.795927    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-sgld6\""
Oct 30 21:52:03.812947 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.812902    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"]
Oct 30 21:52:03.824025 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.824004    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:52:03.825539 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.825518    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:52:03.844772 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.844734    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:03.891176 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.891146    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.891356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.891180    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.891356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.891203    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.891356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.891232    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.891356 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.891347    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.891607 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.891384    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgzc\" (UniqueName: \"kubernetes.io/projected/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-kube-api-access-fjgzc\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.891607 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.891597    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-manifests\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.893748 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.893720    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-tmp\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.893883 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.893844    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-metrics-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.893946 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.893896    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.893946 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.893929    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.949429 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.949356    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgzc\" (UniqueName: \"kubernetes.io/projected/df8e5501-cfc9-4247-bb66-b7e58bc1e97d-kube-api-access-fjgzc\") pod \"opendatahub-operator-controller-manager-7d6c4dc889-w9gzm\" (UID: \"df8e5501-cfc9-4247-bb66-b7e58bc1e97d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:03.994950 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:03.994918    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:04.063314 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:04.063278    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:04.273086 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:04.273017    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:04.458361 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:04.458319    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:04.478012 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:04.477986    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"]
Oct 30 21:52:04.481316 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:52:04.481287    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8e5501_cfc9_4247_bb66_b7e58bc1e97d.slice/crio-722828903f0779d75bf22f26151bb8508b3ee9488d854a7bb0ad03bbec77c97d WatchSource:0}: Error finding container 722828903f0779d75bf22f26151bb8508b3ee9488d854a7bb0ad03bbec77c97d: Status 404 returned error can't find the container with id 722828903f0779d75bf22f26151bb8508b3ee9488d854a7bb0ad03bbec77c97d
Oct 30 21:52:04.945201 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:04.945162    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm" event={"ID":"df8e5501-cfc9-4247-bb66-b7e58bc1e97d","Type":"ContainerStarted","Data":"722828903f0779d75bf22f26151bb8508b3ee9488d854a7bb0ad03bbec77c97d"}
Oct 30 21:52:05.701062 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:05.701017    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:05.925550 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:05.925516    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:07.154870 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:07.154822    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:07.316766 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:07.316730    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:08.706375 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:08.706340    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:08.901990 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:08.901960    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:08.964168 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:08.964134    5503 generic.go:356] "Generic (PLEG): container finished" podID="df8e5501-cfc9-4247-bb66-b7e58bc1e97d" containerID="ebf6b678662aa749a732e13c94a586d807049eb40d479b1d55cfd48b9c425c22" exitCode=0
Oct 30 21:52:08.964329 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:08.964202    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm" event={"ID":"df8e5501-cfc9-4247-bb66-b7e58bc1e97d","Type":"ContainerDied","Data":"ebf6b678662aa749a732e13c94a586d807049eb40d479b1d55cfd48b9c425c22"}
Oct 30 21:52:09.120163 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:09.120132    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:09.970569 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:09.970535    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm" event={"ID":"df8e5501-cfc9-4247-bb66-b7e58bc1e97d","Type":"ContainerStarted","Data":"8a4b6f02ddca80c227b204d9b2b49797abbdac01ea88386090dfd801df3b0a1f"}
Oct 30 21:52:09.971025 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:09.970629    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:10.221616 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:10.221562    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm" podStartSLOduration=2.395530969 podStartE2EDuration="7.221546765s" totalImagesPullingTime="4.345863553s" totalInitContainerRuntime="480.152243ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:10.174550276 +0000 UTC m=+306.509127079" watchObservedRunningTime="2026-10-30 21:52:10.221546765 +0000 UTC m=+306.556123530"
Oct 30 21:52:10.962655 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:10.962608    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:11.633014 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:11.632977    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:11.818439 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:11.818400    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:12.337924 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:12.337892    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:15.305798 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:15.305760    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:15.489754 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:15.489716    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:18.451884 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.451845    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:18.834218 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.834095    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:18.839442 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.839417    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"]
Oct 30 21:52:18.843693 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.843674    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:18.897492 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.897458    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"]
Oct 30 21:52:18.932209 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.932173    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:18.932385 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.932223    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lx4\" (UniqueName: \"kubernetes.io/projected/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-kube-api-access-f5lx4\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:18.932385 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.932348    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:18.943352 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.943321    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:52:18.943352 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.943336    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 21:52:18.943511 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.943359    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:52:18.943511 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.943403    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 21:52:18.943511 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:18.943424    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rmcm6\""
Oct 30 21:52:19.004972 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.004931    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:19.033169 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.033142    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:19.033286 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.033193    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:19.033286 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.033219    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lx4\" (UniqueName: \"kubernetes.io/projected/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-kube-api-access-f5lx4\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:19.033931 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.033914    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:19.035621 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.035600    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:19.091337 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.091275    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lx4\" (UniqueName: \"kubernetes.io/projected/d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520-kube-api-access-f5lx4\") pod \"kuadrant-console-plugin-66c4fb575d-jj7d9\" (UID: \"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:19.172796 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.172770    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"
Oct 30 21:52:19.209942 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.209901    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-k7dzc"]
Oct 30 21:52:19.213746 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.213724    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.267026 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.266986    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:52:19.267026 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.267006    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:52:19.267230 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.267051    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wjw9t\""
Oct 30 21:52:19.267818 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.267799    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-k7dzc"]
Oct 30 21:52:19.336062 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.336036    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8bf3e5-a260-40b2-8332-0e35eca8d793-bound-sa-token\") pod \"cert-manager-597b95cb84-k7dzc\" (UID: \"1e8bf3e5-a260-40b2-8332-0e35eca8d793\") " pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.336219 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.336096    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvdm\" (UniqueName: \"kubernetes.io/projected/1e8bf3e5-a260-40b2-8332-0e35eca8d793-kube-api-access-dgvdm\") pod \"cert-manager-597b95cb84-k7dzc\" (UID: \"1e8bf3e5-a260-40b2-8332-0e35eca8d793\") " pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.351744 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.351668    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:19.436704 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.436677    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvdm\" (UniqueName: \"kubernetes.io/projected/1e8bf3e5-a260-40b2-8332-0e35eca8d793-kube-api-access-dgvdm\") pod \"cert-manager-597b95cb84-k7dzc\" (UID: \"1e8bf3e5-a260-40b2-8332-0e35eca8d793\") " pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.436846 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.436729    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8bf3e5-a260-40b2-8332-0e35eca8d793-bound-sa-token\") pod \"cert-manager-597b95cb84-k7dzc\" (UID: \"1e8bf3e5-a260-40b2-8332-0e35eca8d793\") " pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.494788 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.494758    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e8bf3e5-a260-40b2-8332-0e35eca8d793-bound-sa-token\") pod \"cert-manager-597b95cb84-k7dzc\" (UID: \"1e8bf3e5-a260-40b2-8332-0e35eca8d793\") " pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.495161 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.494950    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvdm\" (UniqueName: \"kubernetes.io/projected/1e8bf3e5-a260-40b2-8332-0e35eca8d793-kube-api-access-dgvdm\") pod \"cert-manager-597b95cb84-k7dzc\" (UID: \"1e8bf3e5-a260-40b2-8332-0e35eca8d793\") " pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.507325 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.507297    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:19.529494 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.529473    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-k7dzc"
Oct 30 21:52:19.536125 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.536102    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9"]
Oct 30 21:52:19.538808 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:52:19.538786    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2c57c9f_1e2a_4f6c_bb53_8dcbfeae6520.slice/crio-7cac258671d326c725847c7f02fa3dbb47cec296d7940ecdc5fc6bb911fc75b3 WatchSource:0}: Error finding container 7cac258671d326c725847c7f02fa3dbb47cec296d7940ecdc5fc6bb911fc75b3: Status 404 returned error can't find the container with id 7cac258671d326c725847c7f02fa3dbb47cec296d7940ecdc5fc6bb911fc75b3
Oct 30 21:52:19.891674 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:19.891628    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-k7dzc"]
Oct 30 21:52:19.894415 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:52:19.894387    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e8bf3e5_a260_40b2_8332_0e35eca8d793.slice/crio-a87a5dab312eb6afe223a4486c4d2d8c5d960fe351e113cb26d007b689ae6e42 WatchSource:0}: Error finding container a87a5dab312eb6afe223a4486c4d2d8c5d960fe351e113cb26d007b689ae6e42: Status 404 returned error can't find the container with id a87a5dab312eb6afe223a4486c4d2d8c5d960fe351e113cb26d007b689ae6e42
Oct 30 21:52:20.007436 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:20.007405    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-k7dzc" event={"ID":"1e8bf3e5-a260-40b2-8332-0e35eca8d793","Type":"ContainerStarted","Data":"a87a5dab312eb6afe223a4486c4d2d8c5d960fe351e113cb26d007b689ae6e42"}
Oct 30 21:52:20.008444 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:20.008422    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9" event={"ID":"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520","Type":"ContainerStarted","Data":"7cac258671d326c725847c7f02fa3dbb47cec296d7940ecdc5fc6bb911fc75b3"}
Oct 30 21:52:20.976956 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:20.976928    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7d6c4dc889-w9gzm"
Oct 30 21:52:21.127237 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:21.127207    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:21.292719 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:21.292625    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:21.692777 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:21.692737    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:21.843326 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:21.843289    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:22.159699 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:22.159660    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:22.318619 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:22.318582    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:23.214710 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:23.214667    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:23.369430 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:23.369344    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:24.030661 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:24.030605    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-k7dzc" event={"ID":"1e8bf3e5-a260-40b2-8332-0e35eca8d793","Type":"ContainerStarted","Data":"da1edeed8586e7b08904b67fbd3967167dfbbeea74856b7b223e8fae1795de72"}
Oct 30 21:52:24.144098 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:24.143996    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-k7dzc" podStartSLOduration=1.204975754 podStartE2EDuration="5.143981885s" totalImagesPullingTime="3.939006131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:24.143374253 +0000 UTC m=+320.477951021" watchObservedRunningTime="2026-10-30 21:52:24.143981885 +0000 UTC m=+320.478558654"
Oct 30 21:52:26.039621 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:26.039585    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9" event={"ID":"d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520","Type":"ContainerStarted","Data":"77956047a7fa7ef6fd88db23ff37b5686e6a87b9df78c8d6bf9281eea53c4ce5"}
Oct 30 21:52:26.149802 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:26.149750    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jj7d9" podStartSLOduration=2.34428777 podStartE2EDuration="8.149736634s" totalImagesPullingTime="5.805448864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:26.148916127 +0000 UTC m=+322.483492896" watchObservedRunningTime="2026-10-30 21:52:26.149736634 +0000 UTC m=+322.484313426"
Oct 30 21:52:26.667238 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:26.667201    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:27.059736 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:27.059656    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:27.254691 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:27.254650    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:28.100011 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:52:28.099974    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:52:33.641152 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:33.641109    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:33.798512 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:33.798481    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:34.047968 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:34.047927    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:34.230067 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:34.230034    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:34.677783 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:34.677746    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:36.132267 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.132231    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:36.182899 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.182862    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"]
Oct 30 21:52:36.187936 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.187913    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.239382 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.239353    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:52:36.283351 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.283320    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373ba953-29a3-4811-aea7-9dd8dfa3b086-tls-certs\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.283525 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.283356    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/373ba953-29a3-4811-aea7-9dd8dfa3b086-tmp\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.283525 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.283392    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jdg5\" (UniqueName: \"kubernetes.io/projected/373ba953-29a3-4811-aea7-9dd8dfa3b086-kube-api-access-4jdg5\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.285862 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.285847    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:52:36.285944 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.285867    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p6chm\""
Oct 30 21:52:36.286829 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.286811    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"]
Oct 30 21:52:36.291394 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.291371    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:36.384488 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.384395    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373ba953-29a3-4811-aea7-9dd8dfa3b086-tls-certs\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.384488 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.384445    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/373ba953-29a3-4811-aea7-9dd8dfa3b086-tmp\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.384488 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.384475    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jdg5\" (UniqueName: \"kubernetes.io/projected/373ba953-29a3-4811-aea7-9dd8dfa3b086-kube-api-access-4jdg5\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.386788 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.386762    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/373ba953-29a3-4811-aea7-9dd8dfa3b086-tmp\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.387033 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.387016    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/373ba953-29a3-4811-aea7-9dd8dfa3b086-tls-certs\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.442470 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.442445    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jdg5\" (UniqueName: \"kubernetes.io/projected/373ba953-29a3-4811-aea7-9dd8dfa3b086-kube-api-access-4jdg5\") pod \"kube-auth-proxy-6d969c5987-wm52k\" (UID: \"373ba953-29a3-4811-aea7-9dd8dfa3b086\") " pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.500589 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.500560    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"
Oct 30 21:52:36.686807 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.686775    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:36.867715 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.867683    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d969c5987-wm52k"]
Oct 30 21:52:36.871853 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:52:36.871823    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373ba953_29a3_4811_aea7_9dd8dfa3b086.slice/crio-50b63eb95913f881cbf29eeab3606498feb41bde7204b5ec784dd344c45ca2ad WatchSource:0}: Error finding container 50b63eb95913f881cbf29eeab3606498feb41bde7204b5ec784dd344c45ca2ad: Status 404 returned error can't find the container with id 50b63eb95913f881cbf29eeab3606498feb41bde7204b5ec784dd344c45ca2ad
Oct 30 21:52:36.901609 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:36.901576    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:37.069093 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:37.069021    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:37.087061 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:37.087029    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k" event={"ID":"373ba953-29a3-4811-aea7-9dd8dfa3b086","Type":"ContainerStarted","Data":"50b63eb95913f881cbf29eeab3606498feb41bde7204b5ec784dd344c45ca2ad"}
Oct 30 21:52:39.030260 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:39.030223    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:39.469171 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:39.469125    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:39.672803 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:39.672759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:40.108860 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:40.108766    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k" event={"ID":"373ba953-29a3-4811-aea7-9dd8dfa3b086","Type":"ContainerStarted","Data":"05511c2b312bac649ee264bc6ee5a4a64f95b71923baf837095d9a5f06b17377"}
Oct 30 21:52:40.222580 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:40.222525    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d969c5987-wm52k" podStartSLOduration=1.288478901 podStartE2EDuration="4.222511898s" totalImagesPullingTime="2.934032997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:40.222133531 +0000 UTC m=+336.556710301" watchObservedRunningTime="2026-10-30 21:52:40.222511898 +0000 UTC m=+336.557088677"
Oct 30 21:52:40.824530 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:40.824490    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:40.976594 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:40.976564    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:43.497976 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.497941    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:43.672719 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.672688    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:52:43.675825 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.675809    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:43.725209 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.725175    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 21:52:43.746962 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.746932    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a7a9e65b-ce49-42f4-93fe-79cf3bc2769e-config-file\") pod \"limitador-limitador-7c9b87d574-x544p\" (UID: \"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:43.747088 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.746978    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znzb\" (UniqueName: \"kubernetes.io/projected/a7a9e65b-ce49-42f4-93fe-79cf3bc2769e-kube-api-access-4znzb\") pod \"limitador-limitador-7c9b87d574-x544p\" (UID: \"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:43.772720 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.772657    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:52:43.848121 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.848092    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a7a9e65b-ce49-42f4-93fe-79cf3bc2769e-config-file\") pod \"limitador-limitador-7c9b87d574-x544p\" (UID: \"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:43.848268 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.848133    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znzb\" (UniqueName: \"kubernetes.io/projected/a7a9e65b-ce49-42f4-93fe-79cf3bc2769e-kube-api-access-4znzb\") pod \"limitador-limitador-7c9b87d574-x544p\" (UID: \"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:43.848753 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.848735    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a7a9e65b-ce49-42f4-93fe-79cf3bc2769e-config-file\") pod \"limitador-limitador-7c9b87d574-x544p\" (UID: \"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:43.906923 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.906891    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znzb\" (UniqueName: \"kubernetes.io/projected/a7a9e65b-ce49-42f4-93fe-79cf3bc2769e-kube-api-access-4znzb\") pod \"limitador-limitador-7c9b87d574-x544p\" (UID: \"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:43.988467 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:43.988439    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:44.005008 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:44.004981    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:52:44.351158 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:44.351086    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:52:44.354269 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:52:44.354242    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a9e65b_ce49_42f4_93fe_79cf3bc2769e.slice/crio-37d6379599bf323b10ed4c0c8ae537963a6907b76ff312090e701e20c6866d2e WatchSource:0}: Error finding container 37d6379599bf323b10ed4c0c8ae537963a6907b76ff312090e701e20c6866d2e: Status 404 returned error can't find the container with id 37d6379599bf323b10ed4c0c8ae537963a6907b76ff312090e701e20c6866d2e
Oct 30 21:52:45.130939 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:45.130903    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p" event={"ID":"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e","Type":"ContainerStarted","Data":"37d6379599bf323b10ed4c0c8ae537963a6907b76ff312090e701e20c6866d2e"}
Oct 30 21:52:48.144508 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:48.144481    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p" event={"ID":"a7a9e65b-ce49-42f4-93fe-79cf3bc2769e","Type":"ContainerStarted","Data":"290f5592fba3d3e9479a19d8d311fa3b544a5ecdc7a3a93425bf6ef0df4d6ed4"}
Oct 30 21:52:48.144856 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:48.144566    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:52:48.255610 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:48.255559    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p" podStartSLOduration=1.5544560920000001 podStartE2EDuration="5.255546044s" totalImagesPullingTime="3.701089952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:48.254814574 +0000 UTC m=+344.589391345" watchObservedRunningTime="2026-10-30 21:52:48.255546044 +0000 UTC m=+344.590122804"
Oct 30 21:52:50.942752 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:50.942711    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.107549 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:51.107516    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.265384 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:51.265290    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:51.416415 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:51.416370    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:52:58.097038 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:52:58.097004    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:52:59.150017 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:52:59.149985    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-x544p"
Oct 30 21:53:06.965080 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:06.965036    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:07.130898 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:07.130862    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:07.824772 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:07.824732    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:08.100976 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:08.100878    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:11.280251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:11.280207    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:28.098404 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:53:28.098311    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:53:28.819679 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:28.819631    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:28.976299 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:28.976266    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:29.133297 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:29.133219    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:29.325075 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:29.325041    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:34.571650 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:34.571614    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:34.742906 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:34.742873    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:37.306824 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.306788    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"]
Oct 30 21:53:37.313142 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.313110    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.365829 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.365741    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 21:53:37.391329 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.391299    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.391506 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.391339    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.391506 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.391462    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44057d31-f693-45ab-99e5-c04ec9b02101-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.391506 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.391498    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffd2\" (UniqueName: \"kubernetes.io/projected/44057d31-f693-45ab-99e5-c04ec9b02101-kube-api-access-sffd2\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.412883 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.412850    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 21:53:37.412883 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.412858    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-j2jxs\""
Oct 30 21:53:37.413081 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.412896    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 21:53:37.413081 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.412909    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:53:37.413081 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.412908    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:53:37.413971 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.413952    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"]
Oct 30 21:53:37.458199 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.458165    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:37.496176 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.496136    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44057d31-f693-45ab-99e5-c04ec9b02101-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.496487 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.496463    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffd2\" (UniqueName: \"kubernetes.io/projected/44057d31-f693-45ab-99e5-c04ec9b02101-kube-api-access-sffd2\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.496606 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.496523    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.496713 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.496565    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.496897 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:53:37.496814    5503 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 21:53:37.497015 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:53:37.496964    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-proxy-source podName:44057d31-f693-45ab-99e5-c04ec9b02101 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:37.996931422 +0000 UTC m=+394.331508197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-wc4d4" (UID: "44057d31-f693-45ab-99e5-c04ec9b02101") : configmap "usage-tenancy-proxy-source" not found
Oct 30 21:53:37.497545 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:53:37.497524    5503 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 21:53:37.497675 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:53:37.497592    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44057d31-f693-45ab-99e5-c04ec9b02101-tls-cert podName:44057d31-f693-45ab-99e5-c04ec9b02101 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:37.997574848 +0000 UTC m=+394.332151608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/44057d31-f693-45ab-99e5-c04ec9b02101-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-wc4d4" (UID: "44057d31-f693-45ab-99e5-c04ec9b02101") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 21:53:37.498528 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.498503    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.555933 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.555897    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffd2\" (UniqueName: \"kubernetes.io/projected/44057d31-f693-45ab-99e5-c04ec9b02101-kube-api-access-sffd2\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:37.623182 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:37.623104    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:38.001599 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:38.001561    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:38.001810 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:38.001671    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44057d31-f693-45ab-99e5-c04ec9b02101-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:38.002434 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:38.002404    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/44057d31-f693-45ab-99e5-c04ec9b02101-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:38.004101 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:38.004079    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44057d31-f693-45ab-99e5-c04ec9b02101-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wc4d4\" (UID: \"44057d31-f693-45ab-99e5-c04ec9b02101\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:38.225156 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:38.225104    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:38.591126 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:38.590981    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"]
Oct 30 21:53:38.594042 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:53:38.594010    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44057d31_f693_45ab_99e5_c04ec9b02101.slice/crio-76f9d8b85a0a1bf873eb5fddd4caba96711c3f7d64eaa2ba0f836e05a30aa7f5 WatchSource:0}: Error finding container 76f9d8b85a0a1bf873eb5fddd4caba96711c3f7d64eaa2ba0f836e05a30aa7f5: Status 404 returned error can't find the container with id 76f9d8b85a0a1bf873eb5fddd4caba96711c3f7d64eaa2ba0f836e05a30aa7f5
Oct 30 21:53:38.596428 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:38.596411    5503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:53:39.360881 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:39.360847    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4" event={"ID":"44057d31-f693-45ab-99e5-c04ec9b02101","Type":"ContainerStarted","Data":"76f9d8b85a0a1bf873eb5fddd4caba96711c3f7d64eaa2ba0f836e05a30aa7f5"}
Oct 30 21:53:51.422547 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:51.422505    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4" event={"ID":"44057d31-f693-45ab-99e5-c04ec9b02101","Type":"ContainerStarted","Data":"be89a57624714fdb76fde1d645c52ad3d9a9a310e325f91740cfed99b43c5a2b"}
Oct 30 21:53:51.422993 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:51.422656    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:53:51.535394 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:51.535290    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4" podStartSLOduration=1.942956012 podStartE2EDuration="14.535272619s" totalImagesPullingTime="12.592316607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:51.533756085 +0000 UTC m=+407.868332849" watchObservedRunningTime="2026-10-30 21:53:51.535272619 +0000 UTC m=+407.869849390"
Oct 30 21:53:56.695728 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:56.695690    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:57.962948 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:57.962914    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:53:58.097549 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:53:58.097514    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:53:58.117864 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:53:58.117837    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:01.431419 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.431381    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"]
Oct 30 21:54:01.435068 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.435050    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.513111 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.513083    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.513298 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.513133    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.513298 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.513187    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.513298 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.513275    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.513465 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.513331    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.513465 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.513362    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.513465 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.513391    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvnj\" (UniqueName: \"kubernetes.io/projected/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-kube-api-access-7hvnj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.533393 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.533364    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"]
Oct 30 21:54:01.614725 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.614692    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.614885 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.614738    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.614885 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.614771    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.614885 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.614804    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.614885 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.614829    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.614885 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.614853    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.614885 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.614877    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvnj\" (UniqueName: \"kubernetes.io/projected/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-kube-api-access-7hvnj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.615566 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.615511    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.617168 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.617142    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.617293 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.617278    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.617410 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.617345    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.617410 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.617360    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.677595 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.677554    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.677761 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.677747    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvnj\" (UniqueName: \"kubernetes.io/projected/ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26-kube-api-access-7hvnj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wjg9j\" (UID: \"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:01.746869 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:01.746808    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:02.175660 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.175597    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"]
Oct 30 21:54:02.178457 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:54:02.178430    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1fe945_0f1b_49dd_ab82_e43fe1ec6d26.slice/crio-0c0cc29a73c787c7601250451b5e252e8a20c27fac965e0e5bd67444ad9d97f6 WatchSource:0}: Error finding container 0c0cc29a73c787c7601250451b5e252e8a20c27fac965e0e5bd67444ad9d97f6: Status 404 returned error can't find the container with id 0c0cc29a73c787c7601250451b5e252e8a20c27fac965e0e5bd67444ad9d97f6
Oct 30 21:54:02.180601 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.180569    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 21:54:02.180718 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.180673    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 21:54:02.432488 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.432415    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wc4d4"
Oct 30 21:54:02.471175 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.471136    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j" event={"ID":"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26","Type":"ContainerStarted","Data":"7ce2419bdd979c05a5f5634ca71fcd7efb085c55948c41a1d447c7a3933e3368"}
Oct 30 21:54:02.471312 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.471200    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:02.471312 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.471222    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j" event={"ID":"ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26","Type":"ContainerStarted","Data":"0c0cc29a73c787c7601250451b5e252e8a20c27fac965e0e5bd67444ad9d97f6"}
Oct 30 21:54:02.660020 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:02.659965    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j" podStartSLOduration=1.6599493490000001 podStartE2EDuration="1.659949349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:02.659064874 +0000 UTC m=+418.993641643" watchObservedRunningTime="2026-10-30 21:54:02.659949349 +0000 UTC m=+418.994526112"
Oct 30 21:54:03.477747 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:03.477594    5503 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wjg9j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:54:03.478243 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:03.477821    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j" podUID="ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:54:06.453816 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.453782    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:06.476330 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.476301    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wjg9j"
Oct 30 21:54:06.603740 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.603708    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:06.662161 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.662132    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"]
Oct 30 21:54:06.662384 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.662358    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" podUID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" containerName="discovery" containerID="cri-o://89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da" gracePeriod=30
Oct 30 21:54:06.715499 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.715413    5503 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-cxp9x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:8080/ready\": dial tcp 10.134.0.20:8080: connect: connection refused" start-of-body=
Oct 30 21:54:06.715499 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.715472    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" podUID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.20:8080/ready\": dial tcp 10.134.0.20:8080: connect: connection refused"
Oct 30 21:54:06.761504 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.761478    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:06.905274 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.905245    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:54:06.916962 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:06.916930    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:07.063424 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063339    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e6d453e-7b20-423c-a96b-a3ab90553e2e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d453e-7b20-423c-a96b-a3ab90553e2e-local-certs\") pod \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") "
Oct 30 21:54:07.063424 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063380    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-token\") pod \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") "
Oct 30 21:54:07.063682 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063440    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-cacerts\") pod \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") "
Oct 30 21:54:07.063682 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063505    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-ca-configmap\") pod \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") "
Oct 30 21:54:07.063682 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063554    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-kube-api-access-plmq7\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-kube-api-access-plmq7\") pod \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") "
Oct 30 21:54:07.063682 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063602    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-dns-cert\") pod \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") "
Oct 30 21:54:07.063682 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063629    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-kubeconfig\") pod \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\" (UID: \"5e6d453e-7b20-423c-a96b-a3ab90553e2e\") "
Oct 30 21:54:07.063977 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.063949    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-ca-configmap" pod "5e6d453e-7b20-423c-a96b-a3ab90553e2e" (UID: "5e6d453e-7b20-423c-a96b-a3ab90553e2e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:07.065875 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.065841    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-dns-cert" pod "5e6d453e-7b20-423c-a96b-a3ab90553e2e" (UID: "5e6d453e-7b20-423c-a96b-a3ab90553e2e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:07.065875 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.065856    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6d453e-7b20-423c-a96b-a3ab90553e2e-local-certs" pod "5e6d453e-7b20-423c-a96b-a3ab90553e2e" (UID: "5e6d453e-7b20-423c-a96b-a3ab90553e2e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:07.066068 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.065997    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-token" pod "5e6d453e-7b20-423c-a96b-a3ab90553e2e" (UID: "5e6d453e-7b20-423c-a96b-a3ab90553e2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:07.066160 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.066144    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-cacerts" pod "5e6d453e-7b20-423c-a96b-a3ab90553e2e" (UID: "5e6d453e-7b20-423c-a96b-a3ab90553e2e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:07.066216 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.066170    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-kube-api-access-plmq7" pod "5e6d453e-7b20-423c-a96b-a3ab90553e2e" (UID: "5e6d453e-7b20-423c-a96b-a3ab90553e2e"). InnerVolumeSpecName "kube-api-access-plmq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:07.066216 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.066168    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-kubeconfig" pod "5e6d453e-7b20-423c-a96b-a3ab90553e2e" (UID: "5e6d453e-7b20-423c-a96b-a3ab90553e2e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:07.075188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.075164    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:07.165343 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.165305    5503 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-ca-configmap\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.165343 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.165337    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmq7\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-kube-api-access-plmq7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.165343 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.165347    5503 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-csr-dns-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.165587 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.165357    5503 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-kubeconfig\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.165587 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.165366    5503 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d453e-7b20-423c-a96b-a3ab90553e2e-local-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.165587 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.165374    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d453e-7b20-423c-a96b-a3ab90553e2e-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.165587 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.165382    5503 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d453e-7b20-423c-a96b-a3ab90553e2e-cacerts\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:07.230007 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.229972    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:07.426902 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.426867    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:07.492346 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.492307    5503 generic.go:356] "Generic (PLEG): container finished" podID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" containerID="89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da" exitCode=0
Oct 30 21:54:07.492850 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.492356    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"
Oct 30 21:54:07.492850 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.492393    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" event={"ID":"5e6d453e-7b20-423c-a96b-a3ab90553e2e","Type":"ContainerDied","Data":"89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da"}
Oct 30 21:54:07.492850 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.492458    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x" event={"ID":"5e6d453e-7b20-423c-a96b-a3ab90553e2e","Type":"ContainerDied","Data":"378d07026225e2f839bc3abad664ac553c4eb45e644e144edf9b8cc920880f63"}
Oct 30 21:54:07.492850 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.492488    5503 scope.go:122] "RemoveContainer" containerID="89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da"
Oct 30 21:54:07.498785 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.498760    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"]
Oct 30 21:54:07.499288 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.499269    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" containerName="discovery"
Oct 30 21:54:07.499351 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.499292    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" containerName="discovery"
Oct 30 21:54:07.499406 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.499393    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" containerName="discovery"
Oct 30 21:54:07.502934 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.502909    5503 scope.go:122] "RemoveContainer" containerID="89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da"
Oct 30 21:54:07.503335 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:54:07.503312    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da\": container with ID starting with 89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da not found: ID does not exist" containerID="89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da"
Oct 30 21:54:07.503412 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.503348    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da"} err="failed to get container status \"89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da\": rpc error: code = NotFound desc = could not find container \"89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da\": container with ID starting with 89a6565b768621dbee52fd6e21fc94027c1be4e82b05ba2689a9c1e7f55596da not found: ID does not exist"
Oct 30 21:54:07.505503 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.505478    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:07.557238 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.557188    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"]
Oct 30 21:54:07.562852 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.562822    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-k5tw2\""
Oct 30 21:54:07.562852 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.562854    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 21:54:07.600709 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.600676    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:07.650942 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.650897    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"]
Oct 30 21:54:07.660614 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.660584    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-cxp9x"]
Oct 30 21:54:07.669498 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.669469    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5h8lq\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:07.669660 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.669502    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5sp\" (UniqueName: \"kubernetes.io/projected/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-kube-api-access-6m5sp\") pod \"llmisvc-controller-manager-869d5dcc5d-5h8lq\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:07.756729 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.756652    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-p67bl"]
Oct 30 21:54:07.759947 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.759924    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.770201 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.770118    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5h8lq\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:07.770324 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.770240    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5sp\" (UniqueName: \"kubernetes.io/projected/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-kube-api-access-6m5sp\") pod \"llmisvc-controller-manager-869d5dcc5d-5h8lq\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:07.772543 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.772514    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5h8lq\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:07.787442 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.787412    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6d453e-7b20-423c-a96b-a3ab90553e2e" path="/var/lib/kubelet/pods/5e6d453e-7b20-423c-a96b-a3ab90553e2e/volumes"
Oct 30 21:54:07.811217 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.811189    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:07.816836 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.816811    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-p67bl"]
Oct 30 21:54:07.835875 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.835844    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5sp\" (UniqueName: \"kubernetes.io/projected/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-kube-api-access-6m5sp\") pod \"llmisvc-controller-manager-869d5dcc5d-5h8lq\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:07.871256 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.871220    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb64a84-3645-47f9-9fb7-f898b1951963-metrics-certs\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.871438 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.871270    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fb64a84-3645-47f9-9fb7-f898b1951963-cert\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.871510 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.871444    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w72b\" (UniqueName: \"kubernetes.io/projected/5fb64a84-3645-47f9-9fb7-f898b1951963-kube-api-access-2w72b\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.892453 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.892419    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 21:54:07.892453 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.892442    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-r9vlz\""
Oct 30 21:54:07.892708 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.892536    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 21:54:07.972149 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.972112    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w72b\" (UniqueName: \"kubernetes.io/projected/5fb64a84-3645-47f9-9fb7-f898b1951963-kube-api-access-2w72b\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.972329 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.972170    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb64a84-3645-47f9-9fb7-f898b1951963-metrics-certs\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.972329 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.972196    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fb64a84-3645-47f9-9fb7-f898b1951963-cert\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.974852 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.974822    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fb64a84-3645-47f9-9fb7-f898b1951963-metrics-certs\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:07.974978 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:07.974822    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5fb64a84-3645-47f9-9fb7-f898b1951963-cert\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:08.030821 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.030756    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w72b\" (UniqueName: \"kubernetes.io/projected/5fb64a84-3645-47f9-9fb7-f898b1951963-kube-api-access-2w72b\") pod \"odh-model-controller-769875db56-p67bl\" (UID: \"5fb64a84-3645-47f9-9fb7-f898b1951963\") " pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:08.072626 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.072596    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:08.119394 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.119358    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:08.439961 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.439933    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-p67bl"]
Oct 30 21:54:08.479899 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:54:08.479875    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb64a84_3645_47f9_9fb7_f898b1951963.slice/crio-143d718f2cbbedb783b7cc3c56f981a7198b3c3f349b270af06157e327f9083e WatchSource:0}: Error finding container 143d718f2cbbedb783b7cc3c56f981a7198b3c3f349b270af06157e327f9083e: Status 404 returned error can't find the container with id 143d718f2cbbedb783b7cc3c56f981a7198b3c3f349b270af06157e327f9083e
Oct 30 21:54:08.489977 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.489953    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"]
Oct 30 21:54:08.492126 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:54:08.492099    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode72e64d4_d87a_4b7d_9170_7f6c9ec2e1da.slice/crio-b0e157d788744e31c5008df3cf55f61390341fc85c8b712bcfc01abe98953a11 WatchSource:0}: Error finding container b0e157d788744e31c5008df3cf55f61390341fc85c8b712bcfc01abe98953a11: Status 404 returned error can't find the container with id b0e157d788744e31c5008df3cf55f61390341fc85c8b712bcfc01abe98953a11
Oct 30 21:54:08.498688 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.498660    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-p67bl" event={"ID":"5fb64a84-3645-47f9-9fb7-f898b1951963","Type":"ContainerStarted","Data":"143d718f2cbbedb783b7cc3c56f981a7198b3c3f349b270af06157e327f9083e"}
Oct 30 21:54:08.499781 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.499761    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq" event={"ID":"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da","Type":"ContainerStarted","Data":"b0e157d788744e31c5008df3cf55f61390341fc85c8b712bcfc01abe98953a11"}
Oct 30 21:54:08.796592 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:08.796512    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:09.861157 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:09.861109    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.022425 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:10.022390    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.178485 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:10.178450    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:10.347554 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:10.347518    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:11.631541 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:11.631510    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:12.520116 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:12.520078    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-p67bl" event={"ID":"5fb64a84-3645-47f9-9fb7-f898b1951963","Type":"ContainerStarted","Data":"c14474ba835f2e97ad9e9745c057df113fcc640b6ca02d75bd538625eec43a04"}
Oct 30 21:54:12.520334 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:12.520181    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:12.521652 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:12.521619    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq" event={"ID":"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da","Type":"ContainerStarted","Data":"b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5"}
Oct 30 21:54:12.521786 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:12.521689    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:12.631802 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:12.631699    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-p67bl" podStartSLOduration=1.749185338 podStartE2EDuration="5.631686235s" totalImagesPullingTime="3.882500897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:12.630808337 +0000 UTC m=+428.965385107" watchObservedRunningTime="2026-10-30 21:54:12.631686235 +0000 UTC m=+428.966263003"
Oct 30 21:54:12.741555 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:12.741451    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq" podStartSLOduration=2.552045357 podStartE2EDuration="5.7414327s" totalImagesPullingTime="3.189387343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:12.740726512 +0000 UTC m=+429.075303282" watchObservedRunningTime="2026-10-30 21:54:12.7414327 +0000 UTC m=+429.076009471"
Oct 30 21:54:14.170952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:14.170921    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:14.324822 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:14.324784    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:14.481352 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:14.481261    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:14.631591 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:14.631552    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:14.791456 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:14.791381    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:14.941725 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:14.941692    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:15.102717 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:15.102624    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:15.255401 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:15.255368    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:15.406939 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:15.406902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:18.909597 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:18.909559    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:54:23.528225 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:23.528195    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-p67bl"
Oct 30 21:54:23.985029 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:23.984976    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:24.178361 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:24.178323    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:54:28.098903 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:54:28.098860    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:54:43.528098 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:54:43.528068    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:54:58.098218 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:54:58.098181    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:55:11.961524 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:11.961491    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:28.097142 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:55:28.097094    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:55:30.329562 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.329529    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-88frl"]
Oct 30 21:55:30.331995 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.331978    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:30.345675 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.345649    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjhk\" (UniqueName: \"kubernetes.io/projected/8d32001d-5d67-45a9-b4f9-1571c4b9d093-kube-api-access-pdjhk\") pod \"llmisvc-controller-manager-6466c554d8-88frl\" (UID: \"8d32001d-5d67-45a9-b4f9-1571c4b9d093\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:30.345794 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.345693    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d32001d-5d67-45a9-b4f9-1571c4b9d093-cert\") pod \"llmisvc-controller-manager-6466c554d8-88frl\" (UID: \"8d32001d-5d67-45a9-b4f9-1571c4b9d093\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:30.387919 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.387884    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-88frl"]
Oct 30 21:55:30.447400 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.447364    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjhk\" (UniqueName: \"kubernetes.io/projected/8d32001d-5d67-45a9-b4f9-1571c4b9d093-kube-api-access-pdjhk\") pod \"llmisvc-controller-manager-6466c554d8-88frl\" (UID: \"8d32001d-5d67-45a9-b4f9-1571c4b9d093\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:30.447400 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.447410    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d32001d-5d67-45a9-b4f9-1571c4b9d093-cert\") pod \"llmisvc-controller-manager-6466c554d8-88frl\" (UID: \"8d32001d-5d67-45a9-b4f9-1571c4b9d093\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:30.449837 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.449810    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d32001d-5d67-45a9-b4f9-1571c4b9d093-cert\") pod \"llmisvc-controller-manager-6466c554d8-88frl\" (UID: \"8d32001d-5d67-45a9-b4f9-1571c4b9d093\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:30.506060 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.506021    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjhk\" (UniqueName: \"kubernetes.io/projected/8d32001d-5d67-45a9-b4f9-1571c4b9d093-kube-api-access-pdjhk\") pod \"llmisvc-controller-manager-6466c554d8-88frl\" (UID: \"8d32001d-5d67-45a9-b4f9-1571c4b9d093\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:30.643082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:30.642995    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:31.007185 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:31.007054    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-88frl"]
Oct 30 21:55:31.009940 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:55:31.009905    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d32001d_5d67_45a9_b4f9_1571c4b9d093.slice/crio-740709c5934f1e03f2a012113e12aa05fe46d6c01377063a9e6ec4b0c87d8667 WatchSource:0}: Error finding container 740709c5934f1e03f2a012113e12aa05fe46d6c01377063a9e6ec4b0c87d8667: Status 404 returned error can't find the container with id 740709c5934f1e03f2a012113e12aa05fe46d6c01377063a9e6ec4b0c87d8667
Oct 30 21:55:31.869839 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:31.869801    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl" event={"ID":"8d32001d-5d67-45a9-b4f9-1571c4b9d093","Type":"ContainerStarted","Data":"3ce5a364b0346d9fe5217b9f0f11b4cac204a34bc6070f0e70f3f885013cc3ef"}
Oct 30 21:55:31.870207 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:31.869866    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl" event={"ID":"8d32001d-5d67-45a9-b4f9-1571c4b9d093","Type":"ContainerStarted","Data":"740709c5934f1e03f2a012113e12aa05fe46d6c01377063a9e6ec4b0c87d8667"}
Oct 30 21:55:31.870207 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:31.869978    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:55:31.980610 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:31.980554    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl" podStartSLOduration=1.219613278 podStartE2EDuration="1.98054019s" totalImagesPullingTime="760.926912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:31.980060348 +0000 UTC m=+508.314637118" watchObservedRunningTime="2026-10-30 21:55:31.98054019 +0000 UTC m=+508.315116970"
Oct 30 21:55:55.457016 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:55:55.456964    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:55:58.097763 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:55:58.097724    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:56:02.877601 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:02.877570    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-88frl"
Oct 30 21:56:03.036385 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.036349    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"]
Oct 30 21:56:03.036600 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.036556    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq" podUID="e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" containerName="manager" containerID="cri-o://b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5" gracePeriod=30
Oct 30 21:56:03.274516 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.274490    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:56:03.419134 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.419062    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-kube-api-access-6m5sp\" (UniqueName: \"kubernetes.io/projected/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-kube-api-access-6m5sp\") pod \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") "
Oct 30 21:56:03.419282 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.419146    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-cert\" (UniqueName: \"kubernetes.io/secret/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-cert\") pod \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\" (UID: \"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da\") "
Oct 30 21:56:03.421293 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.421264    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-cert" pod "e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" (UID: "e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:03.421408 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.421353    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-kube-api-access-6m5sp" pod "e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" (UID: "e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da"). InnerVolumeSpecName "kube-api-access-6m5sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:03.519977 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.519945    5503 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:03.519977 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:03.519974    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m5sp\" (UniqueName: \"kubernetes.io/projected/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da-kube-api-access-6m5sp\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:04.015738 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.015707    5503 generic.go:356] "Generic (PLEG): container finished" podID="e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" containerID="b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5" exitCode=0
Oct 30 21:56:04.016113 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.015760    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"
Oct 30 21:56:04.016113 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.015796    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq" event={"ID":"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da","Type":"ContainerDied","Data":"b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5"}
Oct 30 21:56:04.016113 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.015857    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq" event={"ID":"e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da","Type":"ContainerDied","Data":"b0e157d788744e31c5008df3cf55f61390341fc85c8b712bcfc01abe98953a11"}
Oct 30 21:56:04.016113 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.015887    5503 scope.go:122] "RemoveContainer" containerID="b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5"
Oct 30 21:56:04.026228 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.026213    5503 scope.go:122] "RemoveContainer" containerID="b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5"
Oct 30 21:56:04.026432 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:56:04.026413    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5\": container with ID starting with b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5 not found: ID does not exist" containerID="b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5"
Oct 30 21:56:04.026510 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.026442    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5"} err="failed to get container status \"b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5\": rpc error: code = NotFound desc = could not find container \"b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5\": container with ID starting with b5d90efcc40ca6b29cc1d69043ec47da915965acb561df716c13430bbe9fd3e5 not found: ID does not exist"
Oct 30 21:56:04.149625 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.149543    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:04.174327 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.174297    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"]
Oct 30 21:56:04.178390 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:04.178359    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5h8lq"]
Oct 30 21:56:05.140990 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:05.140955    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:05.788125 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:05.788091    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" path="/var/lib/kubelet/pods/e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da/volumes"
Oct 30 21:56:12.294656 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:12.294593    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:12.528599 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:12.528563    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:12.825082 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:12.825043    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:28.097793 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:56:28.097758    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:56:35.209874 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:35.209841    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:35.755860 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:35.755822    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:37.042689 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:37.042658    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:37.201992 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:37.201959    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:44.058875 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:44.058842    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:48.135169 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:48.135128    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:56:51.211054 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:51.211020    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:54.515188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:54.515139    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:57.720279 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:56:57.720241    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:56:58.098175 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:56:58.098096    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:57:00.933976 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:00.933939    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:01.054596 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:01.054560    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:57:03.855113 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:03.855083    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:57:03.859126 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:03.859102    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 21:57:04.229251 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:04.229213    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:07.990649 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:07.990522    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:11.409422 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:11.409386    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:12.627554 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:12.627524    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:14.373016 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:14.372979    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:57:25.142111 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.142071    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"]
Oct 30 21:57:25.142599 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.142524    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" containerName="manager"
Oct 30 21:57:25.142599 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.142542    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" containerName="manager"
Oct 30 21:57:25.142754 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.142681    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72e64d4-d87a-4b7d-9170-7f6c9ec2e1da" containerName="manager"
Oct 30 21:57:25.145374 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.145353    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.197171 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.197133    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-twq7g\""
Oct 30 21:57:25.227448 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227417    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjdl\" (UniqueName: \"kubernetes.io/projected/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-kube-api-access-chjdl\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227574 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227464    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227574 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227523    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227673 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227629    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227716 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227675    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227770 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227755    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227812 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227778    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227812 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227797    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.227879 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.227821    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.245510 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.245486    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"]
Oct 30 21:57:25.328520 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328496    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.328690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328525    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.328690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328551    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.328690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328566    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.328690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328582    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.328690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328605    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.328690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328661    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjdl\" (UniqueName: \"kubernetes.io/projected/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-kube-api-access-chjdl\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.329015 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328701    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.329015 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.328722    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.329084 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.329042    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.329350 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.329323    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.329445 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.329380    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.329511 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.329443    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.329736 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.329711    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.330903 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.330881    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.331118 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.331101    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.387182 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.387154    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.387340 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.387324    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjdl\" (UniqueName: \"kubernetes.io/projected/bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe-kube-api-access-chjdl\") pod \"maas-default-gateway-openshift-default-6f54779779-ltvq9\" (UID: \"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.460508 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.460472    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:25.886614 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:25.886580    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"]
Oct 30 21:57:26.369370 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:26.369335    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" event={"ID":"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe","Type":"ContainerStarted","Data":"709a0cf19b932fea187fb8c908336172e1aa672b48d5fe7da311fb44050cc1b7"}
Oct 30 21:57:28.096836 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:57:28.096798    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:57:30.416095 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:30.416063    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 21:57:30.416423 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:30.416137    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 21:57:30.416423 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:30.416172    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 21:57:31.391854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:31.391824    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" event={"ID":"bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe","Type":"ContainerStarted","Data":"c975ef448c478a0fad3421ae22560dab95fa9b81a380bccaed460e059a6e08de"}
Oct 30 21:57:31.460758 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:31.460722    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:31.462345 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:31.462319    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ltvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:57:31.462472 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:31.462365    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podUID="bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 21:57:31.567276 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:31.567220    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podStartSLOduration=2.040598825 podStartE2EDuration="6.567201883s" totalImagesPullingTime="4.526603058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:31.553800758 +0000 UTC m=+627.888377541" watchObservedRunningTime="2026-10-30 21:57:31.567201883 +0000 UTC m=+627.901778651"
Oct 30 21:57:32.461441 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:32.461408    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ltvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:57:32.461846 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:32.461459    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podUID="bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 21:57:33.461833 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:33.461798    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ltvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:57:33.462222 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:33.461857    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podUID="bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 21:57:34.461801 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:34.461760    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ltvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:57:34.461983 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:34.461830    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podUID="bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 21:57:35.461384 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:35.461346    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ltvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:57:35.461561 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:35.461418    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podUID="bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 21:57:36.461040 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:36.461000    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ltvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:57:36.461435 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:36.461062    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podUID="bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 21:57:37.460882 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:37.460840    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-ltvq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Oct 30 21:57:37.461039 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:37.460908    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9" podUID="bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Oct 30 21:57:38.615538 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:38.615494    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:38.616043 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:38.615945    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:38.616841 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:38.616807    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-ltvq9"
Oct 30 21:57:39.523773 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:39.523744    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:39.683436 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:39.683401    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:41.160110 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:41.160076    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:41.320828 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:57:41.320798    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:57:58.097732 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:57:58.097693    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:58:12.964851 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:12.964814    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:14.535511 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:14.535476    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:14.708509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:14.708475    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:14.867849 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:14.867775    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:15.115040 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:15.114999    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:15.432594 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:15.432563    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:15.599172 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:15.599139    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:15.846861 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:15.846758    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:16.001675 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:16.001617    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:16.356546 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:16.356517    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:16.551380 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:16.551346    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:16.700713 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:16.700683    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:17.968315 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:17.968280    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:18.211034 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:18.210999    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:19.080476 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:19.080433    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:19.243029 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:19.242995    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:19.484549 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:19.484515    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:19.674760 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:19.674727    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:19.866897 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:19.866817    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:20.038201 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:20.038165    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:21.747212 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:21.747180    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.012061 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:22.011990    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.278117 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:22.278035    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.446048 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:22.446016    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.604601 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:22.604521    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:22.759779 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:22.759746    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:23.246130 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:23.246096    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:23.446407 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:23.446375    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:24.791563 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:24.791526    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:25.000256 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:25.000222    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:25.169286 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:25.169254    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:25.333755 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:25.333722    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:27.419466 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:27.419435    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:27.594574 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:27.594537    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:28.098170 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:58:28.098135    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:58:28.639818 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:28.639783    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:28.801079 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:28.801047    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:28.981911 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:28.981878    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:29.884099 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:29.884057    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:30.048476 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:30.048434    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:30.203842 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:30.203805    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:30.356248 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:30.356217    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:32.589197 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:32.589165    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:58:32.686822 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:32.686790    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:32.904676 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:32.904628    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:34.773670 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:34.773623    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:34.927772 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:34.927743    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:35.110548 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:35.110469    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:37.871340 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:37.871306    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:38.062398 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.062353    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:38.425400 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.425365    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"]
Oct 30 21:58:38.428919 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.428900    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:38.525171 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.525111    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4705e99c\""
Oct 30 21:58:38.525171 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.525117    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4705e99c-dockercfg-769x4\""
Oct 30 21:58:38.526387 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.526370    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"]
Oct 30 21:58:38.602929 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.602895    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhw6k\" (UniqueName: \"kubernetes.io/projected/55c73cd2-19fb-4409-9163-bf6b21e3cead-kube-api-access-lhw6k\") pod \"payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") " pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:38.603087 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.602961    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c73cd2-19fb-4409-9163-bf6b21e3cead-plugins-config-volume\") pod \"payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") " pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:38.703976 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.703906    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhw6k\" (UniqueName: \"kubernetes.io/projected/55c73cd2-19fb-4409-9163-bf6b21e3cead-kube-api-access-lhw6k\") pod \"payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") " pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:38.703976 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.703957    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c73cd2-19fb-4409-9163-bf6b21e3cead-plugins-config-volume\") pod \"payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") " pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:38.704682 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.704665    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c73cd2-19fb-4409-9163-bf6b21e3cead-plugins-config-volume\") pod \"payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") " pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:38.762444 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:38.762411    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhw6k\" (UniqueName: \"kubernetes.io/projected/55c73cd2-19fb-4409-9163-bf6b21e3cead-kube-api-access-lhw6k\") pod \"payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") " pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:39.041907 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:39.041814    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:39.176351 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:39.176314    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:39.329201 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:39.329090    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:39.407168 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:39.407144    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"]
Oct 30 21:58:39.410031 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:58:39.410003    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c73cd2_19fb_4409_9163_bf6b21e3cead.slice/crio-4abe8e4044d4e297a38eafbaf7a2ad1f90d1564d02fd133eba2f6caa1579178f WatchSource:0}: Error finding container 4abe8e4044d4e297a38eafbaf7a2ad1f90d1564d02fd133eba2f6caa1579178f: Status 404 returned error can't find the container with id 4abe8e4044d4e297a38eafbaf7a2ad1f90d1564d02fd133eba2f6caa1579178f
Oct 30 21:58:39.412129 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:39.412110    5503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:58:39.500330 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:39.500299    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:39.689349 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:39.689317    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q" event={"ID":"55c73cd2-19fb-4409-9163-bf6b21e3cead","Type":"ContainerStarted","Data":"4abe8e4044d4e297a38eafbaf7a2ad1f90d1564d02fd133eba2f6caa1579178f"}
Oct 30 21:58:41.443225 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:41.443196    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:58:43.708288 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:43.708250    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q" event={"ID":"55c73cd2-19fb-4409-9163-bf6b21e3cead","Type":"ContainerStarted","Data":"e9e998d0478e06df86b4a18bb52c51dc119f3f06b812ade48ee4d1951af33bc6"}
Oct 30 21:58:43.708828 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:43.708358    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:43.818942 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:43.818895    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q" podStartSLOduration=2.37681255 podStartE2EDuration="5.818881766s" totalImagesPullingTime="3.442069216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:43.818265884 +0000 UTC m=+700.152842654" watchObservedRunningTime="2026-10-30 21:58:43.818881766 +0000 UTC m=+700.153458577"
Oct 30 21:58:44.086501 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:44.086420    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:48.729537 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:48.729490    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:53.350751 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:53.350709    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:53.510289 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:53.510246    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:53.961619 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:53.961577    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:54.716528 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:54.716499    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:58:55.439003 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.438970    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"]
Oct 30 21:58:55.442353 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.442335    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.492158 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.492135    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:55.492158 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.492146    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 21:58:55.539105 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.539076    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:58:55.539268 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.539140    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:58:55.540239 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.540222    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"]
Oct 30 21:58:55.562406 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.562379    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.562562 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.562420    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvpn\" (UniqueName: \"kubernetes.io/projected/a96bd18f-de2a-46c0-8263-c7bc7243bc67-kube-api-access-grvpn\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.562614 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.562572    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.663196 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.663155    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.663369 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.663201    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvpn\" (UniqueName: \"kubernetes.io/projected/a96bd18f-de2a-46c0-8263-c7bc7243bc67-kube-api-access-grvpn\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.663369 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.663247    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.663911 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.663888    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.665543 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.665522    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.721271 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.721205    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvpn\" (UniqueName: \"kubernetes.io/projected/a96bd18f-de2a-46c0-8263-c7bc7243bc67-kube-api-access-grvpn\") pod \"maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:55.754430 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:55.754394    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:58:56.120787 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:56.120763    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"]
Oct 30 21:58:56.123265 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:58:56.123232    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96bd18f_de2a_46c0_8263_c7bc7243bc67.slice/crio-df8af5ba6f17e2ae236c14e6e8491f98804f60818544d21381a7d728b39e7bbc WatchSource:0}: Error finding container df8af5ba6f17e2ae236c14e6e8491f98804f60818544d21381a7d728b39e7bbc: Status 404 returned error can't find the container with id df8af5ba6f17e2ae236c14e6e8491f98804f60818544d21381a7d728b39e7bbc
Oct 30 21:58:56.763699 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:56.763662    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s" event={"ID":"a96bd18f-de2a-46c0-8263-c7bc7243bc67","Type":"ContainerStarted","Data":"df8af5ba6f17e2ae236c14e6e8491f98804f60818544d21381a7d728b39e7bbc"}
Oct 30 21:58:58.097045 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:58:58.097003    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:58:58.661075 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:58.661046    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:58:58.838982 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:58:58.838955    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:59:01.788653 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:01.788334    5503 generic.go:356] "Generic (PLEG): container finished" podID="a96bd18f-de2a-46c0-8263-c7bc7243bc67" containerID="091b6cae3b4589023e700a87d6ac8da7cb5147d11b7cbc1807b847fc2b50415f" exitCode=0
Oct 30 21:59:01.790121 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:01.790099    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s" event={"ID":"a96bd18f-de2a-46c0-8263-c7bc7243bc67","Type":"ContainerDied","Data":"091b6cae3b4589023e700a87d6ac8da7cb5147d11b7cbc1807b847fc2b50415f"}
Oct 30 21:59:02.920797 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:02.920771    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:59:03.026760 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.026730    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-xks-ca\") pod \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") "
Oct 30 21:59:03.026907 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.026783    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-openshift-ca\") pod \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") "
Oct 30 21:59:03.026907 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.026860    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96bd18f-de2a-46c0-8263-c7bc7243bc67-kube-api-access-grvpn\" (UniqueName: \"kubernetes.io/projected/a96bd18f-de2a-46c0-8263-c7bc7243bc67-kube-api-access-grvpn\") pod \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\" (UID: \"a96bd18f-de2a-46c0-8263-c7bc7243bc67\") "
Oct 30 21:59:03.027171 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.027147    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-openshift-ca" pod "a96bd18f-de2a-46c0-8263-c7bc7243bc67" (UID: "a96bd18f-de2a-46c0-8263-c7bc7243bc67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:03.028970 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.028948    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-xks-ca" pod "a96bd18f-de2a-46c0-8263-c7bc7243bc67" (UID: "a96bd18f-de2a-46c0-8263-c7bc7243bc67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:03.029068 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.029017    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96bd18f-de2a-46c0-8263-c7bc7243bc67-kube-api-access-grvpn" pod "a96bd18f-de2a-46c0-8263-c7bc7243bc67" (UID: "a96bd18f-de2a-46c0-8263-c7bc7243bc67"). InnerVolumeSpecName "kube-api-access-grvpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:03.128332 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.128246    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvpn\" (UniqueName: \"kubernetes.io/projected/a96bd18f-de2a-46c0-8263-c7bc7243bc67-kube-api-access-grvpn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:03.128332 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.128279    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:03.128332 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.128290    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96bd18f-de2a-46c0-8263-c7bc7243bc67-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:03.401249 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.401217    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:03.798462 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.798431    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"
Oct 30 21:59:03.798594 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.798459    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s" event={"ID":"a96bd18f-de2a-46c0-8263-c7bc7243bc67","Type":"ContainerDied","Data":"df8af5ba6f17e2ae236c14e6e8491f98804f60818544d21381a7d728b39e7bbc"}
Oct 30 21:59:03.798594 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:03.798501    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df8af5ba6f17e2ae236c14e6e8491f98804f60818544d21381a7d728b39e7bbc"
Oct 30 21:59:08.364626 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:08.364590    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:10.561129 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:10.561094    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:10.750987 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:10.750954    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:11.124143 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.124106    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"]
Oct 30 21:59:11.124504 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.124490    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a96bd18f-de2a-46c0-8263-c7bc7243bc67" containerName="revoke-keys"
Oct 30 21:59:11.124549 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.124505    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96bd18f-de2a-46c0-8263-c7bc7243bc67" containerName="revoke-keys"
Oct 30 21:59:11.124582 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.124576    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96bd18f-de2a-46c0-8263-c7bc7243bc67" containerName="revoke-keys"
Oct 30 21:59:11.127854 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.127837    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.201195 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.201156    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.201195 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.201195    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sdx6\" (UniqueName: \"kubernetes.io/projected/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-kube-api-access-8sdx6\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.201396 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.201295    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.226094 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.226056    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:11.226094 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.226071    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:59:11.226094 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.226086    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:59:11.226337 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.226146    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 21:59:11.227293 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.227273    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"]
Oct 30 21:59:11.302804 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.302771    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.302804 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.302805    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sdx6\" (UniqueName: \"kubernetes.io/projected/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-kube-api-access-8sdx6\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.303055 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.302857    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.303576 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.303551    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.305355 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.305329    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.361509 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.361485    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sdx6\" (UniqueName: \"kubernetes.io/projected/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-kube-api-access-8sdx6\") pod \"maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.440724 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.440680    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:11.814876 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.814746    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"]
Oct 30 21:59:11.817866 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:59:11.817841    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d98a19_b93a_409f_9d45_f8d9cb7dcd56.slice/crio-559b39911b5d069033313d0dcb4df84ed9c8f259e58840018419b3aebc7d6d27 WatchSource:0}: Error finding container 559b39911b5d069033313d0dcb4df84ed9c8f259e58840018419b3aebc7d6d27: Status 404 returned error can't find the container with id 559b39911b5d069033313d0dcb4df84ed9c8f259e58840018419b3aebc7d6d27
Oct 30 21:59:11.832914 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:11.832885    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7" event={"ID":"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56","Type":"ContainerStarted","Data":"559b39911b5d069033313d0dcb4df84ed9c8f259e58840018419b3aebc7d6d27"}
Oct 30 21:59:12.838653 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:12.838607    5503 generic.go:356] "Generic (PLEG): container finished" podID="b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" containerID="dff25c5dbbd7170ddc1526956a91e743ac1b10b0c15560bf7ef1b1378751b425" exitCode=0
Oct 30 21:59:12.839038 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:12.838696    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7" event={"ID":"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56","Type":"ContainerDied","Data":"dff25c5dbbd7170ddc1526956a91e743ac1b10b0c15560bf7ef1b1378751b425"}
Oct 30 21:59:13.315748 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:13.315715    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:13.545511 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:13.545462    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:13.753936 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:13.753904    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:59:13.969112 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:13.969090    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:14.131974 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.131890    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-xks-ca\") pod \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") "
Oct 30 21:59:14.132150 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.132036    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-openshift-ca\") pod \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") "
Oct 30 21:59:14.132150 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.132102    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-kube-api-access-8sdx6\" (UniqueName: \"kubernetes.io/projected/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-kube-api-access-8sdx6\") pod \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\" (UID: \"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56\") "
Oct 30 21:59:14.132454 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.132428    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-openshift-ca" pod "b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" (UID: "b5d98a19-b93a-409f-9d45-f8d9cb7dcd56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:14.133952 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.133932    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-xks-ca" pod "b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" (UID: "b5d98a19-b93a-409f-9d45-f8d9cb7dcd56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:14.134218 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.134205    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-kube-api-access-8sdx6" pod "b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" (UID: "b5d98a19-b93a-409f-9d45-f8d9cb7dcd56"). InnerVolumeSpecName "kube-api-access-8sdx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:14.232727 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.232693    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sdx6\" (UniqueName: \"kubernetes.io/projected/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-kube-api-access-8sdx6\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:14.232727 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.232722    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:14.232727 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.232736    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:14.848397 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.848365    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7" event={"ID":"b5d98a19-b93a-409f-9d45-f8d9cb7dcd56","Type":"ContainerDied","Data":"559b39911b5d069033313d0dcb4df84ed9c8f259e58840018419b3aebc7d6d27"}
Oct 30 21:59:14.848575 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.848419    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="559b39911b5d069033313d0dcb4df84ed9c8f259e58840018419b3aebc7d6d27"
Oct 30 21:59:14.848575 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:14.848375    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"
Oct 30 21:59:18.050171 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:18.050133    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:19.516763 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:19.516732    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:19.669995 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:19.669952    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:19.825549 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:19.825458    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:19.985704 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:19.985665    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:20.035986 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:20.035955    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"]
Oct 30 21:59:20.082921 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:20.082851    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63d97fec-5nt7s"]
Oct 30 21:59:20.167246 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:20.167203    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:21.002338 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:21.002303    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:21.687285 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:21.687251    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:21.788503 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:21.788469    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96bd18f-de2a-46c0-8263-c7bc7243bc67" path="/var/lib/kubelet/pods/a96bd18f-de2a-46c0-8263-c7bc7243bc67/volumes"
Oct 30 21:59:21.943729 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:21.943627    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:22.380959 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:22.380882    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:22.554369 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:22.554318    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:24.241358 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:24.241326    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:25.409358 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:25.409324    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:25.688575 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:25.688486    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:25.901445 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:25.901416    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:26.377575 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:26.377542    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:26.559720 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:26.559683    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:27.274408 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:27.274373    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:27.453386 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:27.453353    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:27.616529 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:27.616440    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:27.770821 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:27.770787    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"]
Oct 30 21:59:27.817316 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:27.817281    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4b2e37ae-cmxd7"]
Oct 30 21:59:27.935712 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:27.935662    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:28.097714 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:59:28.097682    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 21:59:28.355402 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:28.355321    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:28.785209 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:28.785174    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:28.943702 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:28.943673    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:29.105750 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.105671    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:29.300690 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.300649    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:29.519395 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.519361    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"]
Oct 30 21:59:29.519755 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.519740    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" containerName="revoke-keys"
Oct 30 21:59:29.519853 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.519755    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" containerName="revoke-keys"
Oct 30 21:59:29.519853 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.519844    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" containerName="revoke-keys"
Oct 30 21:59:29.523188 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.523168    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.574383 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.574350    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.574537 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.574418    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ftt\" (UniqueName: \"kubernetes.io/projected/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-kube-api-access-29ftt\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.574537 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.574500    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.575974 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.575947    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 21:59:29.577730 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.577704    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"]
Oct 30 21:59:29.623039 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.623013    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:29.623291 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.623273    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:59:29.623397 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.623293    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:59:29.675401 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.675372    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.675582 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.675423    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ftt\" (UniqueName: \"kubernetes.io/projected/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-kube-api-access-29ftt\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.675582 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.675453    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.676043 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.676018    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.677885 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.677863    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.694822 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.694791    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:29.740087 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.740053    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ftt\" (UniqueName: \"kubernetes.io/projected/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-kube-api-access-29ftt\") pod \"maas-api-revoke-keys-e2e-derive-4705e99c-sznf2\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:29.786995 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.786924    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d98a19-b93a-409f-9d45-f8d9cb7dcd56" path="/var/lib/kubelet/pods/b5d98a19-b93a-409f-9d45-f8d9cb7dcd56/volumes"
Oct 30 21:59:29.836047 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:29.836015    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:30.076474 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:30.076386    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:30.203329 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:30.203307    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"]
Oct 30 21:59:30.205708 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:59:30.205680    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdd2ccb_f27c_47fa_b1f5_bb51bfb4f1e9.slice/crio-9549cd57a7c369ecf505f0b92f0118286d8518705189b5fb5dd968b7d22a216f WatchSource:0}: Error finding container 9549cd57a7c369ecf505f0b92f0118286d8518705189b5fb5dd968b7d22a216f: Status 404 returned error can't find the container with id 9549cd57a7c369ecf505f0b92f0118286d8518705189b5fb5dd968b7d22a216f
Oct 30 21:59:30.922342 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:30.922304    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2" event={"ID":"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9","Type":"ContainerStarted","Data":"39c12999be69fdca65b9fcb140964c63895e74fc41e69b1abc5e27a485afbf94"}
Oct 30 21:59:30.922846 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:30.922365    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2" event={"ID":"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9","Type":"ContainerStarted","Data":"9549cd57a7c369ecf505f0b92f0118286d8518705189b5fb5dd968b7d22a216f"}
Oct 30 21:59:31.033148 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:31.033089    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2" podStartSLOduration=2.033074573 podStartE2EDuration="2.033074573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:31.032430725 +0000 UTC m=+747.367007495" watchObservedRunningTime="2026-10-30 21:59:31.033074573 +0000 UTC m=+747.367651333"
Oct 30 21:59:31.928879 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:31.928827    5503 generic.go:356] "Generic (PLEG): container finished" podID="7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" containerID="39c12999be69fdca65b9fcb140964c63895e74fc41e69b1abc5e27a485afbf94" exitCode=0
Oct 30 21:59:31.928879 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:31.928882    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2" event={"ID":"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9","Type":"ContainerDied","Data":"39c12999be69fdca65b9fcb140964c63895e74fc41e69b1abc5e27a485afbf94"}
Oct 30 21:59:32.106872 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:32.106834    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:32.543113 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:32.543075    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:32.801720 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:32.801621    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:32.867825 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:32.867790    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:59:33.075193 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.075118    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:33.102057 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.102022    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-xks-ca\") pod \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") "
Oct 30 21:59:33.102238 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.102087    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-openshift-ca\") pod \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") "
Oct 30 21:59:33.102238 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.102121    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-kube-api-access-29ftt\" (UniqueName: \"kubernetes.io/projected/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-kube-api-access-29ftt\") pod \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\" (UID: \"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9\") "
Oct 30 21:59:33.102461 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.102436    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-openshift-ca" pod "7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" (UID: "7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:33.104180 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.104153    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-xks-ca" pod "7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" (UID: "7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:33.104276 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.104257    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-kube-api-access-29ftt" pod "7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" (UID: "7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9"). InnerVolumeSpecName "kube-api-access-29ftt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:33.202955 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.202919    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:33.202955 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.202951    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:33.202955 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.202961    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ftt\" (UniqueName: \"kubernetes.io/projected/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9-kube-api-access-29ftt\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:33.939693 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.939665    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"
Oct 30 21:59:33.939836 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.939664    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2" event={"ID":"7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9","Type":"ContainerDied","Data":"9549cd57a7c369ecf505f0b92f0118286d8518705189b5fb5dd968b7d22a216f"}
Oct 30 21:59:33.939836 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.939783    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9549cd57a7c369ecf505f0b92f0118286d8518705189b5fb5dd968b7d22a216f"
Oct 30 21:59:33.959904 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:33.959876    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:34.124338 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:34.124262    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:34.290901 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:34.290873    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:34.555788 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:34.555760    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:34.762129 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:34.762099    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:34.925958 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:34.925922    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:37.085575 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:37.085543    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:38.981066 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:38.981033    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:39.155951 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:39.155919    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:40.638601 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:40.638570    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:40.801283 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:40.801246    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:41.603821 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:41.603788    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:41.790102 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:41.790068    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:41.954057 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:41.954023    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:42.119792 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.119758    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:42.158190 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.158159    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"]
Oct 30 21:59:42.158570 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.158540    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" containerName="revoke-keys"
Oct 30 21:59:42.158570 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.158566    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" containerName="revoke-keys"
Oct 30 21:59:42.158805 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.158665    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" containerName="revoke-keys"
Oct 30 21:59:42.161941 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.161917    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.261573 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.261495    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d09fff\""
Oct 30 21:59:42.261573 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.261497    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d09fff-dockercfg-bzvnh\""
Oct 30 21:59:42.262596 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.262548    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"]
Oct 30 21:59:42.280383 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.280347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:42.287506 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.287481    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545a76bc-1f02-4365-a785-fe0062df7c20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.287584 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.287521    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jd4\" (UniqueName: \"kubernetes.io/projected/545a76bc-1f02-4365-a785-fe0062df7c20-kube-api-access-g8jd4\") pod \"payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.388381 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.388345    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545a76bc-1f02-4365-a785-fe0062df7c20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.388381 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.388389    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jd4\" (UniqueName: \"kubernetes.io/projected/545a76bc-1f02-4365-a785-fe0062df7c20-kube-api-access-g8jd4\") pod \"payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.389154 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.389121    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545a76bc-1f02-4365-a785-fe0062df7c20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.446977 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.446949    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jd4\" (UniqueName: \"kubernetes.io/projected/545a76bc-1f02-4365-a785-fe0062df7c20-kube-api-access-g8jd4\") pod \"payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.478625 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.478596    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:42.801224 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.801122    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"]
Oct 30 21:59:42.843631 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.843607    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"]
Oct 30 21:59:42.846175 ip-10-0-141-86 kubenswrapper[5503]: W1030 21:59:42.846147    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545a76bc_1f02_4365_a785_fe0062df7c20.slice/crio-fc917d3cedb25668367a268bed00b9407bf75ac3c3d8f43c08bf59b484c82694 WatchSource:0}: Error finding container fc917d3cedb25668367a268bed00b9407bf75ac3c3d8f43c08bf59b484c82694: Status 404 returned error can't find the container with id fc917d3cedb25668367a268bed00b9407bf75ac3c3d8f43c08bf59b484c82694
Oct 30 21:59:42.981309 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.981276    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" event={"ID":"545a76bc-1f02-4365-a785-fe0062df7c20","Type":"ContainerStarted","Data":"15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031"}
Oct 30 21:59:42.981504 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.981328    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" event={"ID":"545a76bc-1f02-4365-a785-fe0062df7c20","Type":"ContainerStarted","Data":"fc917d3cedb25668367a268bed00b9407bf75ac3c3d8f43c08bf59b484c82694"}
Oct 30 21:59:42.981504 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:42.981358    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:43.092894 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:43.092808    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" podStartSLOduration=1.092794656 podStartE2EDuration="1.092794656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:43.091557072 +0000 UTC m=+759.426133842" watchObservedRunningTime="2026-10-30 21:59:43.092794656 +0000 UTC m=+759.427371424"
Oct 30 21:59:43.194953 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:43.194915    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:43.424548 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:43.424514    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:46.050013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:46.049981    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:47.449027 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:47.448990    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:47.705918 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:47.705836    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:47.802001 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:47.801965    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q" podUID="55c73cd2-19fb-4409-9163-bf6b21e3cead" containerName="payload-processing" containerID="cri-o://e9e998d0478e06df86b4a18bb52c51dc119f3f06b812ade48ee4d1951af33bc6" gracePeriod=25
Oct 30 21:59:47.856747 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:47.856719    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:48.006408 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.006375    5503 generic.go:356] "Generic (PLEG): container finished" podID="55c73cd2-19fb-4409-9163-bf6b21e3cead" containerID="e9e998d0478e06df86b4a18bb52c51dc119f3f06b812ade48ee4d1951af33bc6" exitCode=0
Oct 30 21:59:48.006567 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.006455    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q" event={"ID":"55c73cd2-19fb-4409-9163-bf6b21e3cead","Type":"ContainerDied","Data":"e9e998d0478e06df86b4a18bb52c51dc119f3f06b812ade48ee4d1951af33bc6"}
Oct 30 21:59:48.034013 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.033988    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:59:48.138519 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.138489    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c73cd2-19fb-4409-9163-bf6b21e3cead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c73cd2-19fb-4409-9163-bf6b21e3cead-plugins-config-volume\") pod \"55c73cd2-19fb-4409-9163-bf6b21e3cead\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") "
Oct 30 21:59:48.138705 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.138667    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c73cd2-19fb-4409-9163-bf6b21e3cead-kube-api-access-lhw6k\" (UniqueName: \"kubernetes.io/projected/55c73cd2-19fb-4409-9163-bf6b21e3cead-kube-api-access-lhw6k\") pod \"55c73cd2-19fb-4409-9163-bf6b21e3cead\" (UID: \"55c73cd2-19fb-4409-9163-bf6b21e3cead\") "
Oct 30 21:59:48.138945 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.138920    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c73cd2-19fb-4409-9163-bf6b21e3cead-plugins-config-volume" pod "55c73cd2-19fb-4409-9163-bf6b21e3cead" (UID: "55c73cd2-19fb-4409-9163-bf6b21e3cead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:48.140650 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.140618    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c73cd2-19fb-4409-9163-bf6b21e3cead-kube-api-access-lhw6k" pod "55c73cd2-19fb-4409-9163-bf6b21e3cead" (UID: "55c73cd2-19fb-4409-9163-bf6b21e3cead"). InnerVolumeSpecName "kube-api-access-lhw6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:48.240279 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.240214    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhw6k\" (UniqueName: \"kubernetes.io/projected/55c73cd2-19fb-4409-9163-bf6b21e3cead-kube-api-access-lhw6k\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:48.240279 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:48.240239    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c73cd2-19fb-4409-9163-bf6b21e3cead-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:49.011884 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:49.011851    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q" event={"ID":"55c73cd2-19fb-4409-9163-bf6b21e3cead","Type":"ContainerDied","Data":"4abe8e4044d4e297a38eafbaf7a2ad1f90d1564d02fd133eba2f6caa1579178f"}
Oct 30 21:59:49.012291 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:49.011905    5503 scope.go:122] "RemoveContainer" containerID="e9e998d0478e06df86b4a18bb52c51dc119f3f06b812ade48ee4d1951af33bc6"
Oct 30 21:59:49.012291 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:49.011857    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"
Oct 30 21:59:49.166917 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:49.166884    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"]
Oct 30 21:59:49.214075 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:49.214044    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4705e99c-6b46b75947-sll7q"]
Oct 30 21:59:49.563474 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:49.563442    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:49.786951 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:49.786923    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c73cd2-19fb-4409-9163-bf6b21e3cead" path="/var/lib/kubelet/pods/55c73cd2-19fb-4409-9163-bf6b21e3cead/volumes"
Oct 30 21:59:50.441055 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:50.441018    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:51.092839 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:51.092803    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:51.384555 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:51.384477    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 21:59:51.402848 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:51.402820    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:52.576841 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:52.576810    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:52.774765 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:52.774732    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:52.930991 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:52.930962    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:53.988436 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:53.988411    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 21:59:55.560699 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:55.560665    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:55.720686 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:55.720650    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:55.890847 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:55.890768    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:57.702835 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:57.702802    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:57.852764 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:57.852731    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:58.009908 ip-10-0-141-86 kubenswrapper[5503]: I1030 21:59:58.009826    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 21:59:58.098443 ip-10-0-141-86 kubenswrapper[5503]: E1030 21:59:58.098406    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:00:00.202289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.202256    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"]
Oct 30 22:00:00.202695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.202614    5503 cpu_manager.go:414] "containerMap: removing container" podUID="55c73cd2-19fb-4409-9163-bf6b21e3cead" containerName="payload-processing"
Oct 30 22:00:00.202695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.202626    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c73cd2-19fb-4409-9163-bf6b21e3cead" containerName="payload-processing"
Oct 30 22:00:00.202695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.202697    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c73cd2-19fb-4409-9163-bf6b21e3cead" containerName="payload-processing"
Oct 30 22:00:00.207140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.207122    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.212192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.212168    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"]
Oct 30 22:00:00.215378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.215362    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.258593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.258574    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:00:00.258716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.258657    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:00:00.305697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.305674    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:00:00.307781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.307756    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"]
Oct 30 22:00:00.312051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.312027    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"]
Oct 30 22:00:00.356010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.355979    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hp59\" (UniqueName: \"kubernetes.io/projected/2f08538e-4260-457d-b214-fb49583b3093-kube-api-access-9hp59\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.356151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.356039    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4645200f-8fc7-4bf9-9269-d891c439e42e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.356202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.356146    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrw2\" (UniqueName: \"kubernetes.io/projected/4645200f-8fc7-4bf9-9269-d891c439e42e-kube-api-access-2nrw2\") pod \"maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.356202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.356175    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f08538e-4260-457d-b214-fb49583b3093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.456679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.456589    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrw2\" (UniqueName: \"kubernetes.io/projected/4645200f-8fc7-4bf9-9269-d891c439e42e-kube-api-access-2nrw2\") pod \"maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.456679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.456621    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f08538e-4260-457d-b214-fb49583b3093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.456679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.456676    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hp59\" (UniqueName: \"kubernetes.io/projected/2f08538e-4260-457d-b214-fb49583b3093-kube-api-access-9hp59\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.456877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.456723    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4645200f-8fc7-4bf9-9269-d891c439e42e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.457300 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.457282    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f08538e-4260-457d-b214-fb49583b3093-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.457339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.457328    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4645200f-8fc7-4bf9-9269-d891c439e42e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.514539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.514508    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrw2\" (UniqueName: \"kubernetes.io/projected/4645200f-8fc7-4bf9-9269-d891c439e42e-kube-api-access-2nrw2\") pod \"maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.514893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.514874    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hp59\" (UniqueName: \"kubernetes.io/projected/2f08538e-4260-457d-b214-fb49583b3093-kube-api-access-9hp59\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.519677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.519660    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:00.527842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.527820    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:00.893819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.893790    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"]
Oct 30 22:00:00.896202 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:00:00.896173    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice/crio-62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436 WatchSource:0}: Error finding container 62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436: Status 404 returned error can't find the container with id 62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436
Oct 30 22:00:00.941435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:00.941410    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"]
Oct 30 22:00:00.943746 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:00:00.943719    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-99aae12a24980ec9746a7b1cee3d8b988b9177d9bfab26e5574766516649012f WatchSource:0}: Error finding container 99aae12a24980ec9746a7b1cee3d8b988b9177d9bfab26e5574766516649012f: Status 404 returned error can't find the container with id 99aae12a24980ec9746a7b1cee3d8b988b9177d9bfab26e5574766516649012f
Oct 30 22:00:01.073975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:01.073935    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" event={"ID":"2f08538e-4260-457d-b214-fb49583b3093","Type":"ContainerStarted","Data":"62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436"}
Oct 30 22:00:01.075021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:01.074998    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp" event={"ID":"4645200f-8fc7-4bf9-9269-d891c439e42e","Type":"ContainerStarted","Data":"99aae12a24980ec9746a7b1cee3d8b988b9177d9bfab26e5574766516649012f"}
Oct 30 22:00:01.271202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:01.271171    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:00:01.802017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:01.801986    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:02.028059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.027993    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:02.080646 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.080600    5503 generic.go:356] "Generic (PLEG): container finished" podID="2f08538e-4260-457d-b214-fb49583b3093" containerID="f52e19e5a6f1a0b91ca45ee50a23d5c520b62972a799ae4ba0bff48584c5abbc" exitCode=7
Oct 30 22:00:02.080810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.080663    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" event={"ID":"2f08538e-4260-457d-b214-fb49583b3093","Type":"ContainerDied","Data":"f52e19e5a6f1a0b91ca45ee50a23d5c520b62972a799ae4ba0bff48584c5abbc"}
Oct 30 22:00:02.080922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.080907    5503 scope.go:122] "RemoveContainer" containerID="f52e19e5a6f1a0b91ca45ee50a23d5c520b62972a799ae4ba0bff48584c5abbc"
Oct 30 22:00:02.082821 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.082801    5503 generic.go:356] "Generic (PLEG): container finished" podID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerID="0272574c428c8c62ccff8a2d948ed585ca21f45d56a67e47ad6cbb9cb7831f83" exitCode=60
Oct 30 22:00:02.082920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.082826    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp" event={"ID":"4645200f-8fc7-4bf9-9269-d891c439e42e","Type":"ContainerDied","Data":"0272574c428c8c62ccff8a2d948ed585ca21f45d56a67e47ad6cbb9cb7831f83"}
Oct 30 22:00:02.083059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.083043    5503 scope.go:122] "RemoveContainer" containerID="0272574c428c8c62ccff8a2d948ed585ca21f45d56a67e47ad6cbb9cb7831f83"
Oct 30 22:00:02.173417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.173386    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"]
Oct 30 22:00:02.195223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.195180    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:02.346017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.345991    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"]
Oct 30 22:00:02.351151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.351126    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:02.403296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.403261    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e524ce-dockercfg-hrlb5\""
Oct 30 22:00:02.404245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.404219    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e524ce\""
Oct 30 22:00:02.404763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.404735    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"]
Oct 30 22:00:02.476781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.476750    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwfj5\" (UniqueName: \"kubernetes.io/projected/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-kube-api-access-vwfj5\") pod \"payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:02.476948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.476790    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:02.577606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.577566    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwfj5\" (UniqueName: \"kubernetes.io/projected/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-kube-api-access-vwfj5\") pod \"payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:02.577793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.577621    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:02.578820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.578790    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:02.636169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.636094    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwfj5\" (UniqueName: \"kubernetes.io/projected/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-kube-api-access-vwfj5\") pod \"payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:02.662769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:02.662742    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:03.028059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.028020    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"]
Oct 30 22:00:03.030337 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:00:03.030307    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9bcd58_5e7f_482b_b61e_06fcd5882a4b.slice/crio-52b0468e9a939deea518166a17a43c65c8013100ef3d46a7fb4112a5d1434b9f WatchSource:0}: Error finding container 52b0468e9a939deea518166a17a43c65c8013100ef3d46a7fb4112a5d1434b9f: Status 404 returned error can't find the container with id 52b0468e9a939deea518166a17a43c65c8013100ef3d46a7fb4112a5d1434b9f
Oct 30 22:00:03.088521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.088496    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp" event={"ID":"4645200f-8fc7-4bf9-9269-d891c439e42e","Type":"ContainerStarted","Data":"64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17"}
Oct 30 22:00:03.088672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.088532    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerName="cleanup" containerID="cri-o://64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17" gracePeriod=30
Oct 30 22:00:03.090200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.090174    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" event={"ID":"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b","Type":"ContainerStarted","Data":"f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf"}
Oct 30 22:00:03.090319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.090234    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:03.090319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.090261    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" event={"ID":"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b","Type":"ContainerStarted","Data":"52b0468e9a939deea518166a17a43c65c8013100ef3d46a7fb4112a5d1434b9f"}
Oct 30 22:00:03.092184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.092159    5503 generic.go:356] "Generic (PLEG): container finished" podID="2f08538e-4260-457d-b214-fb49583b3093" containerID="386bb2bb358fad53d4c3e1445225f4a94180368dec048f0b5d34e93fa9455092" exitCode=7
Oct 30 22:00:03.092286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.092192    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" event={"ID":"2f08538e-4260-457d-b214-fb49583b3093","Type":"ContainerDied","Data":"386bb2bb358fad53d4c3e1445225f4a94180368dec048f0b5d34e93fa9455092"}
Oct 30 22:00:03.092286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.092227    5503 scope.go:122] "RemoveContainer" containerID="f52e19e5a6f1a0b91ca45ee50a23d5c520b62972a799ae4ba0bff48584c5abbc"
Oct 30 22:00:03.092431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.092413    5503 scope.go:122] "RemoveContainer" containerID="386bb2bb358fad53d4c3e1445225f4a94180368dec048f0b5d34e93fa9455092"
Oct 30 22:00:03.092646 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:03.092612    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n_odh-ai-gateway-infra(2f08538e-4260-457d-b214-fb49583b3093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" podUID="2f08538e-4260-457d-b214-fb49583b3093"
Oct 30 22:00:03.208123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.208015    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp" podStartSLOduration=3.207998737 podStartE2EDuration="3.207998737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.206793081 +0000 UTC m=+779.541369851" watchObservedRunningTime="2026-10-30 22:00:03.207998737 +0000 UTC m=+779.542575505"
Oct 30 22:00:03.436910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.436857    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" podStartSLOduration=1.4368432580000001 podStartE2EDuration="1.436843258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.436122954 +0000 UTC m=+779.770699722" watchObservedRunningTime="2026-10-30 22:00:03.436843258 +0000 UTC m=+779.771420026"
Oct 30 22:00:03.659827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.659796    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:03.908587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:03.908536    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:04.071276 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:04.071175    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:04.097930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:04.097905    5503 scope.go:122] "RemoveContainer" containerID="386bb2bb358fad53d4c3e1445225f4a94180368dec048f0b5d34e93fa9455092"
Oct 30 22:00:04.098137 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:04.098116    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n_odh-ai-gateway-infra(2f08538e-4260-457d-b214-fb49583b3093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" podUID="2f08538e-4260-457d-b214-fb49583b3093"
Oct 30 22:00:04.147827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:04.147778    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"]
Oct 30 22:00:04.152316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:04.152284    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4705e99c-sznf2"]
Oct 30 22:00:04.280014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:04.279985    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:05.169701 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:05.169651    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:05.789100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:05.789068    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9" path="/var/lib/kubelet/pods/7cdd2ccb-f27c-47fa-b1f5-bb51bfb4f1e9/volumes"
Oct 30 22:00:05.817300 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:05.817264    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:06.332287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:06.332248    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:09.014305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:09.014273    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:10.568974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:10.568944    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:10.841727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:10.841649    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:11.033782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:11.033751    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:11.174564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:11.174532    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:00:11.230681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:11.230628    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:11.731343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:11.731304    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:11.912087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:11.912056    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:13.638625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:13.638592    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:14.100526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:14.100500    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:00:14.166048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:14.166008    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:15.066269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.066235    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"]
Oct 30 22:00:15.071039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.071020    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.167792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.167760    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:15.170388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.170369    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"default-dockercfg-v8p5t\""
Oct 30 22:00:15.170523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.170390    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"simulator-w2-7f66ab-kserve-self-signed-certs\""
Oct 30 22:00:15.170592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.170551    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"openshift-service-ca.crt\""
Oct 30 22:00:15.170705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.170687    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"kube-root-ca.crt\""
Oct 30 22:00:15.171860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.171839    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"]
Oct 30 22:00:15.189094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.189071    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-home\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.189200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.189106    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tls-certs\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.189200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.189132    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-dshm\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.189274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.189259    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tmp-dir\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.189311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.189283    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-model-cache\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.189311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.189305    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkl8\" (UniqueName: \"kubernetes.io/projected/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-kube-api-access-qpkl8\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.290447 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.290407    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkl8\" (UniqueName: \"kubernetes.io/projected/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-kube-api-access-qpkl8\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.290615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.290514    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-home\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.290615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.290538    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tls-certs\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.290615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.290563    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-dshm\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.290615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.290597    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tmp-dir\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.290615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.290615    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-model-cache\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.291048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.291028    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-model-cache\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.291129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.291109    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-home\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.291172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.291129    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tmp-dir\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.293106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.293077    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-dshm\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.293523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.293259    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tls-certs\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.348817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.348749    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkl8\" (UniqueName: \"kubernetes.io/projected/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-kube-api-access-qpkl8\") pod \"simulator-w2-7f66ab-kserve-596dcf649f-nft4z\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") " pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.382352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.382317    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:15.757466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.757435    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"]
Oct 30 22:00:15.787780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:15.787760    5503 scope.go:122] "RemoveContainer" containerID="386bb2bb358fad53d4c3e1445225f4a94180368dec048f0b5d34e93fa9455092"
Oct 30 22:00:16.158697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:16.158663    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" event={"ID":"954db4d4-4b0e-461f-aa6c-9b71d2ed6340","Type":"ContainerStarted","Data":"d4fdf67a1fb2a758f1626e122ebed5e195f82959b7a2393b2fbc0c69fadc3cfe"}
Oct 30 22:00:16.160422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:16.160394    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" event={"ID":"2f08538e-4260-457d-b214-fb49583b3093","Type":"ContainerStarted","Data":"18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba"}
Oct 30 22:00:16.274624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:16.274540    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" podStartSLOduration=16.27452754 podStartE2EDuration="16.27452754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:16.274089174 +0000 UTC m=+792.608665944" watchObservedRunningTime="2026-10-30 22:00:16.27452754 +0000 UTC m=+792.609104308"
Oct 30 22:00:17.167398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:17.167364    5503 generic.go:356] "Generic (PLEG): container finished" podID="2f08538e-4260-457d-b214-fb49583b3093" containerID="18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba" exitCode=60
Oct 30 22:00:17.167832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:17.167439    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" event={"ID":"2f08538e-4260-457d-b214-fb49583b3093","Type":"ContainerDied","Data":"18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba"}
Oct 30 22:00:17.167832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:17.167503    5503 scope.go:122] "RemoveContainer" containerID="386bb2bb358fad53d4c3e1445225f4a94180368dec048f0b5d34e93fa9455092"
Oct 30 22:00:17.167947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:17.167862    5503 scope.go:122] "RemoveContainer" containerID="18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba"
Oct 30 22:00:17.168102 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:17.168083    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n_odh-ai-gateway-infra(2f08538e-4260-457d-b214-fb49583b3093)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" podUID="2f08538e-4260-457d-b214-fb49583b3093"
Oct 30 22:00:17.283317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:17.283282    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"]
Oct 30 22:00:17.621521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:17.621437    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:17.829393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:17.829364    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:18.359828 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:18.359793    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:18.414384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:18.414347    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f08538e-4260-457d-b214-fb49583b3093-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f08538e-4260-457d-b214-fb49583b3093-maas-api-service-ca\") pod \"2f08538e-4260-457d-b214-fb49583b3093\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") "
Oct 30 22:00:18.414557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:18.414514    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f08538e-4260-457d-b214-fb49583b3093-kube-api-access-9hp59\" (UniqueName: \"kubernetes.io/projected/2f08538e-4260-457d-b214-fb49583b3093-kube-api-access-9hp59\") pod \"2f08538e-4260-457d-b214-fb49583b3093\" (UID: \"2f08538e-4260-457d-b214-fb49583b3093\") "
Oct 30 22:00:18.414781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:18.414753    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f08538e-4260-457d-b214-fb49583b3093-maas-api-service-ca" pod "2f08538e-4260-457d-b214-fb49583b3093" (UID: "2f08538e-4260-457d-b214-fb49583b3093"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:18.417941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:18.417909    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f08538e-4260-457d-b214-fb49583b3093-kube-api-access-9hp59" pod "2f08538e-4260-457d-b214-fb49583b3093" (UID: "2f08538e-4260-457d-b214-fb49583b3093"). InnerVolumeSpecName "kube-api-access-9hp59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:18.515676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:18.515582    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hp59\" (UniqueName: \"kubernetes.io/projected/2f08538e-4260-457d-b214-fb49583b3093-kube-api-access-9hp59\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:18.515676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:18.515614    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f08538e-4260-457d-b214-fb49583b3093-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.185042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.185004    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n" event={"ID":"2f08538e-4260-457d-b214-fb49583b3093","Type":"ContainerDied","Data":"62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436"}
Oct 30 22:00:19.185042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.185032    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"
Oct 30 22:00:19.185318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.185080    5503 scope.go:122] "RemoveContainer" containerID="18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba"
Oct 30 22:00:19.204140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.204117    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.357243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.357220    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"]
Oct 30 22:00:19.374154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.374130    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889960-x9d7n"]
Oct 30 22:00:19.386273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.386249    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.547511 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.547440    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:19.788233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:19.788201    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f08538e-4260-457d-b214-fb49583b3093" path="/var/lib/kubelet/pods/2f08538e-4260-457d-b214-fb49583b3093/volumes"
Oct 30 22:00:20.190721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:20.190676    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" event={"ID":"954db4d4-4b0e-461f-aa6c-9b71d2ed6340","Type":"ContainerStarted","Data":"3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa"}
Oct 30 22:00:20.303978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:20.303918    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podStartSLOduration=1.764933962 podStartE2EDuration="5.303902414s" totalImagesPullingTime="3.538968452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:20.302213621 +0000 UTC m=+796.636790390" watchObservedRunningTime="2026-10-30 22:00:20.303902414 +0000 UTC m=+796.638479184"
Oct 30 22:00:20.382880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:20.382847    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:23.190114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:23.190074    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:23.343550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:23.343518    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:24.350088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:24.350046    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:24.511873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:24.511842    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:24.936345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:24.936296    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:25.097215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:25.097178    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:25.270340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:25.270231    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:25.383436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:25.383399    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:25.396002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:25.395978    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:25.770584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:25.770544    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:26.237867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:26.237840    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:00:27.962763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:27.962728    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:00:28.106815 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:28.106773    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:00:29.505337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:29.505303    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:33.146629 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:00:33.146593    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice/crio-18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba.scope WatchSource:0}: Error finding container 18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba: Status 404 returned error can't find the container with id 18adc069ab24088cf13429d200a50c88e0c4d7f6a54041203c39ba92a599daba
Oct 30 22:00:33.160459 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:33.160399    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice/crio-62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-conmon-64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:33.161967 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:33.161842    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-conmon-64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice/crio-62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:33.162092 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:33.161986    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice/crio-62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:33.163909 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:33.163868    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-99aae12a24980ec9746a7b1cee3d8b988b9177d9bfab26e5574766516649012f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645200f_8fc7_4bf9_9269_d891c439e42e.slice/crio-conmon-64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f08538e_4260_457d_b214_fb49583b3093.slice/crio-62974f51b66fecff7c1f728fd3323c91eff6537a0f7d721cca657d6d26159436\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:33.255249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.255218    5503 generic.go:356] "Generic (PLEG): container finished" podID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerID="64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17" exitCode=137
Oct 30 22:00:33.255413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.255267    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp" event={"ID":"4645200f-8fc7-4bf9-9269-d891c439e42e","Type":"ContainerDied","Data":"64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17"}
Oct 30 22:00:33.255413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.255373    5503 scope.go:122] "RemoveContainer" containerID="0272574c428c8c62ccff8a2d948ed585ca21f45d56a67e47ad6cbb9cb7831f83"
Oct 30 22:00:33.295236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.295211    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:33.348078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.348014    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4645200f-8fc7-4bf9-9269-d891c439e42e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4645200f-8fc7-4bf9-9269-d891c439e42e-maas-api-service-ca\") pod \"4645200f-8fc7-4bf9-9269-d891c439e42e\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") "
Oct 30 22:00:33.348078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.348050    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4645200f-8fc7-4bf9-9269-d891c439e42e-kube-api-access-2nrw2\" (UniqueName: \"kubernetes.io/projected/4645200f-8fc7-4bf9-9269-d891c439e42e-kube-api-access-2nrw2\") pod \"4645200f-8fc7-4bf9-9269-d891c439e42e\" (UID: \"4645200f-8fc7-4bf9-9269-d891c439e42e\") "
Oct 30 22:00:33.348351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.348328    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4645200f-8fc7-4bf9-9269-d891c439e42e-maas-api-service-ca" pod "4645200f-8fc7-4bf9-9269-d891c439e42e" (UID: "4645200f-8fc7-4bf9-9269-d891c439e42e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:33.350093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.350072    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4645200f-8fc7-4bf9-9269-d891c439e42e-kube-api-access-2nrw2" pod "4645200f-8fc7-4bf9-9269-d891c439e42e" (UID: "4645200f-8fc7-4bf9-9269-d891c439e42e"). InnerVolumeSpecName "kube-api-access-2nrw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:33.448801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.448771    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4645200f-8fc7-4bf9-9269-d891c439e42e-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:33.448801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:33.448798    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nrw2\" (UniqueName: \"kubernetes.io/projected/4645200f-8fc7-4bf9-9269-d891c439e42e-kube-api-access-2nrw2\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:34.261797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.261764    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp" event={"ID":"4645200f-8fc7-4bf9-9269-d891c439e42e","Type":"ContainerDied","Data":"99aae12a24980ec9746a7b1cee3d8b988b9177d9bfab26e5574766516649012f"}
Oct 30 22:00:34.262267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.261824    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"
Oct 30 22:00:34.262267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.261828    5503 scope.go:122] "RemoveContainer" containerID="64212bac35ea520993f6f6bdcb946b7fc8d64a29571951b9d344e707e91b3a17"
Oct 30 22:00:34.407741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.407707    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:34.502052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.502020    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"]
Oct 30 22:00:34.506475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.506450    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-4705e99c-29889960-rpxtp"]
Oct 30 22:00:34.559707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.559620    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:34.715600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:34.715564    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:35.269032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:35.268990    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:35.429467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:35.429418    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:35.601419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:35.601328    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:35.787356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:35.787314    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" path="/var/lib/kubelet/pods/4645200f-8fc7-4bf9-9269-d891c439e42e/volumes"
Oct 30 22:00:36.493185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.492884    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"]
Oct 30 22:00:36.493708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493595    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerName="cleanup"
Oct 30 22:00:36.493708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493615    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerName="cleanup"
Oct 30 22:00:36.493708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493627    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.493708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493649    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.493708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493665    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.493708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493672    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.494024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493760    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.494024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493774    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.494024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.493949    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.494170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.494003    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerName="cleanup"
Oct 30 22:00:36.494170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.494041    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:36.494270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.494189    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerName="cleanup"
Oct 30 22:00:36.500076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.500049    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.549586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.549540    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e524ce-openshift-default-dockercfg-zs996\""
Oct 30 22:00:36.573897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.573871    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.573917    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.573984    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289cdaae-6028-42e7-a615-c4358d99287e-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.574005    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289cdaae-6028-42e7-a615-c4358d99287e-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.574028    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9lp\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-kube-api-access-vk9lp\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.574070    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.574170    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.574199    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.574296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.574284    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675084    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675121    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675143    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675187    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675234    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675266    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289cdaae-6028-42e7-a615-c4358d99287e-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675289    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289cdaae-6028-42e7-a615-c4358d99287e-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675321    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9lp\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-kube-api-access-vk9lp\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675388    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675555    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675621    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675737    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.675955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675933    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289cdaae-6028-42e7-a615-c4358d99287e-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.676008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.675972    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.677466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.677445    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.677724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.677707    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289cdaae-6028-42e7-a615-c4358d99287e-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.688893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.688868    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"]
Oct 30 22:00:36.732880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.732852    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9lp\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-kube-api-access-vk9lp\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.733065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.733047    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:36.812713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:36.812677    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:37.237353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:37.237329    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"]
Oct 30 22:00:37.239520 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:00:37.239489    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289cdaae_6028_42e7_a615_c4358d99287e.slice/crio-9177bcc665a8d8a2279f899ae92de161b617d3e76fb0310ba7bddef0b0ef3c2f WatchSource:0}: Error finding container 9177bcc665a8d8a2279f899ae92de161b617d3e76fb0310ba7bddef0b0ef3c2f: Status 404 returned error can't find the container with id 9177bcc665a8d8a2279f899ae92de161b617d3e76fb0310ba7bddef0b0ef3c2f
Oct 30 22:00:37.241927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:37.241890    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:00:37.242038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:37.241975    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:00:37.242102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:37.242047    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:00:37.279782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:37.279753    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" event={"ID":"289cdaae-6028-42e7-a615-c4358d99287e","Type":"ContainerStarted","Data":"9177bcc665a8d8a2279f899ae92de161b617d3e76fb0310ba7bddef0b0ef3c2f"}
Oct 30 22:00:38.286809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:38.286776    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" event={"ID":"289cdaae-6028-42e7-a615-c4358d99287e","Type":"ContainerStarted","Data":"7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc"}
Oct 30 22:00:38.451704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:38.451607    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podStartSLOduration=2.451587133 podStartE2EDuration="2.451587133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:38.450793267 +0000 UTC m=+814.785370038" watchObservedRunningTime="2026-10-30 22:00:38.451587133 +0000 UTC m=+814.786163904"
Oct 30 22:00:38.813222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:38.813192    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:38.814512 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:38.814483    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:00:38.814660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:38.814527    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:00:39.163402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:39.163362    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:39.814024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:39.813989    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:00:39.814444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:39.814039    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:00:40.336075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.333980    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"]
Oct 30 22:00:40.336075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.334545    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerName="cleanup"
Oct 30 22:00:40.336075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.334564    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4645200f-8fc7-4bf9-9269-d891c439e42e" containerName="cleanup"
Oct 30 22:00:40.336075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.334675    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f08538e-4260-457d-b214-fb49583b3093" containerName="cleanup"
Oct 30 22:00:40.338095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.337804    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.407337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.407306    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-model-cache\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.407506 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.407437    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tmp-dir\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.407689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.407664    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-home\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.407866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.407844    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tls-certs\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.407971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.407888    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-dshm\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.408053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.408038    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbp7v\" (UniqueName: \"kubernetes.io/projected/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-kube-api-access-wbp7v\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.438324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.438172    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b47e8d\"/\"default-dockercfg-kvds8\""
Oct 30 22:00:40.438324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.438284    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b47e8d\"/\"simulator-w3-b47e8d-kserve-self-signed-certs\""
Oct 30 22:00:40.442342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.442316    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"]
Oct 30 22:00:40.442807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.442772    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b47e8d\"/\"openshift-service-ca.crt\""
Oct 30 22:00:40.442887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.442862    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b47e8d\"/\"kube-root-ca.crt\""
Oct 30 22:00:40.480462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.480436    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:40.508626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.508602    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tmp-dir\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.508740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.508661    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-home\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.508838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.508819    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tls-certs\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.508889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.508869    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-dshm\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.508927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.508909    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbp7v\" (UniqueName: \"kubernetes.io/projected/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-kube-api-access-wbp7v\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.508967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.508950    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tmp-dir\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.509003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.508954    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-model-cache\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.509036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.509004    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-home\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.509255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.509229    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-model-cache\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.511005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.510982    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-dshm\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.511254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.511237    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tls-certs\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.567679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.567646    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbp7v\" (UniqueName: \"kubernetes.io/projected/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-kube-api-access-wbp7v\") pod \"simulator-w3-b47e8d-kserve-5d64874f84-jbqwf\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") " pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.651950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.651926    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:40.813686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.813652    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:00:40.813847 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:40.813704    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:00:41.026852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:41.026759    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"]
Oct 30 22:00:41.042769 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:00:41.042728    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8c62843_bc04_4d6a_a63e_ca97b65f9fc4.slice/crio-4fb2202ec7247e22160423abad7b3a020aaeddfbfb6b96588608733e91685f4b WatchSource:0}: Error finding container 4fb2202ec7247e22160423abad7b3a020aaeddfbfb6b96588608733e91685f4b: Status 404 returned error can't find the container with id 4fb2202ec7247e22160423abad7b3a020aaeddfbfb6b96588608733e91685f4b
Oct 30 22:00:41.301908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:41.301826    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" event={"ID":"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4","Type":"ContainerStarted","Data":"679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf"}
Oct 30 22:00:41.301908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:41.301868    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" event={"ID":"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4","Type":"ContainerStarted","Data":"4fb2202ec7247e22160423abad7b3a020aaeddfbfb6b96588608733e91685f4b"}
Oct 30 22:00:41.414699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:41.414618    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podStartSLOduration=1.414605648 podStartE2EDuration="1.414605648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:41.413852985 +0000 UTC m=+817.748429754" watchObservedRunningTime="2026-10-30 22:00:41.414605648 +0000 UTC m=+817.749182416"
Oct 30 22:00:41.652377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:41.652349    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:41.813225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:41.813192    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:00:41.813384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:41.813244    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:00:42.813988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:42.813950    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 22:00:42.814383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:42.814004    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 22:00:43.817958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:43.817926    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:43.818405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:43.818223    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:43.818906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:43.818890    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:00:43.865793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:43.865759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:45.254353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:45.254318    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:00:48.569062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:48.569027    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:50.652232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:50.652185    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:50.664429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:50.664406    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:51.370106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:51.370071    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:00:53.231480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.231445    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:53.523191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.523119    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"]
Oct 30 22:00:53.528912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.528891    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.627719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.627681    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"simulator-w0-d09fff-kserve-self-signed-certs\""
Oct 30 22:00:53.627719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.627720    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"openshift-service-ca.crt\""
Oct 30 22:00:53.627932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.627721    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"kube-root-ca.crt\""
Oct 30 22:00:53.627932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.627761    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"default-dockercfg-wq5ll\""
Oct 30 22:00:53.628810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.628784    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"]
Oct 30 22:00:53.633447 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.633421    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:53.658653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.658605    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tls-certs\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.658653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.658653    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j878k\" (UniqueName: \"kubernetes.io/projected/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-kube-api-access-j878k\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.658808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.658673    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-home\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.658808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.658777    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-model-cache\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.658879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.658819    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-dshm\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.658879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.658850    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tmp-dir\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.759849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.759811    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tls-certs\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.759849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.759856    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j878k\" (UniqueName: \"kubernetes.io/projected/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-kube-api-access-j878k\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.760126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.759873    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-home\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.760126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.759902    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-model-cache\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.760126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.759947    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-dshm\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.760126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.759980    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tmp-dir\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.760340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.760322    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-home\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.760421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.760399    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tmp-dir\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.760501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.760470    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-model-cache\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.762339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.762309    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-dshm\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.762499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.762478    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tls-certs\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.822128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.822052    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j878k\" (UniqueName: \"kubernetes.io/projected/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-kube-api-access-j878k\") pod \"simulator-w0-d09fff-kserve-7bfdff55d-crtgf\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") " pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:53.840715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:53.840682    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:54.217298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:54.217273    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"]
Oct 30 22:00:54.219619 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:00:54.219595    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11159a97_55c0_4e97_8bb5_1a0ae53ca1fe.slice/crio-773f26201ff39d00498e9b5e796c7b8b3a91152c188fe063b097f9075e11d72b WatchSource:0}: Error finding container 773f26201ff39d00498e9b5e796c7b8b3a91152c188fe063b097f9075e11d72b: Status 404 returned error can't find the container with id 773f26201ff39d00498e9b5e796c7b8b3a91152c188fe063b097f9075e11d72b
Oct 30 22:00:54.374907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:54.374867    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" event={"ID":"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe","Type":"ContainerStarted","Data":"d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1"}
Oct 30 22:00:54.374907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:54.374913    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" event={"ID":"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe","Type":"ContainerStarted","Data":"773f26201ff39d00498e9b5e796c7b8b3a91152c188fe063b097f9075e11d72b"}
Oct 30 22:00:54.488857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:54.488760    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podStartSLOduration=1.4887468 podStartE2EDuration="1.4887468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:54.487945373 +0000 UTC m=+830.822522206" watchObservedRunningTime="2026-10-30 22:00:54.4887468 +0000 UTC m=+830.823323569"
Oct 30 22:00:54.841004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:54.840916    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:00:56.771729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:56.771688    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:57.352835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:57.352795    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:57.531707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:57.531668    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:57.694330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:57.694297    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:58.098219 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:00:58.098137    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:00:58.255913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:58.255873    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:00:58.642403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:00:58.642322    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:01:02.910700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:02.910662    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:03.841836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:03.841806    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:01:03.854057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:03.854033    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:01:04.432662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:04.432618    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:01:07.640582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:07.640545    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:09.831555 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.831519    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"]
Oct 30 22:01:09.836694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.836670    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:09.891314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.891276    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"]
Oct 30 22:01:09.916948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.916915    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e93339-64a5-4beb-a491-557f5d4506e0-tls-certs\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:09.917128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.916978    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-home\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:09.917128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.917092    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-tmp-dir\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:09.917236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.917145    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-model-cache\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:09.917236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.917172    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-dshm\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:09.917236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.917196    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74cn9\" (UniqueName: \"kubernetes.io/projected/e2e93339-64a5-4beb-a491-557f5d4506e0-kube-api-access-74cn9\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:09.925791 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.925761    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:09.936525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:09.936507    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"premium-w2-7f66ab-kserve-self-signed-certs\""
Oct 30 22:01:10.018542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.018499    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-home\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.018775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.018550    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-tmp-dir\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.018775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.018582    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-model-cache\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.018775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.018605    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-dshm\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.018775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.018668    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74cn9\" (UniqueName: \"kubernetes.io/projected/e2e93339-64a5-4beb-a491-557f5d4506e0-kube-api-access-74cn9\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.019006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.018819    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e93339-64a5-4beb-a491-557f5d4506e0-tls-certs\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.019006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.018990    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-home\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.019094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.019019    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-model-cache\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.019094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.019051    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-tmp-dir\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.020982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.020958    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-dshm\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.021230 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.021212    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e93339-64a5-4beb-a491-557f5d4506e0-tls-certs\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.077316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.077278    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74cn9\" (UniqueName: \"kubernetes.io/projected/e2e93339-64a5-4beb-a491-557f5d4506e0-kube-api-access-74cn9\") pod \"premium-w2-7f66ab-kserve-5b9884c8db-qgnc8\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") " pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.148332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.148242    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:10.521483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:10.521452    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"]
Oct 30 22:01:10.523778 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:01:10.523750    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2e93339_64a5_4beb_a491_557f5d4506e0.slice/crio-5cc032441c7bb3cb5f16b0e8c28d7ac953b5062bcc7d278cfb24c28bfc6575da WatchSource:0}: Error finding container 5cc032441c7bb3cb5f16b0e8c28d7ac953b5062bcc7d278cfb24c28bfc6575da: Status 404 returned error can't find the container with id 5cc032441c7bb3cb5f16b0e8c28d7ac953b5062bcc7d278cfb24c28bfc6575da
Oct 30 22:01:11.455459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:11.455422    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" event={"ID":"e2e93339-64a5-4beb-a491-557f5d4506e0","Type":"ContainerStarted","Data":"a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6"}
Oct 30 22:01:11.455889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:11.455465    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" event={"ID":"e2e93339-64a5-4beb-a491-557f5d4506e0","Type":"ContainerStarted","Data":"5cc032441c7bb3cb5f16b0e8c28d7ac953b5062bcc7d278cfb24c28bfc6575da"}
Oct 30 22:01:11.570749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:11.570378    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podStartSLOduration=2.570359582 podStartE2EDuration="2.570359582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:11.570063638 +0000 UTC m=+847.904640409" watchObservedRunningTime="2026-10-30 22:01:11.570359582 +0000 UTC m=+847.904936354"
Oct 30 22:01:11.701398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:11.701369    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.148416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:12.148378    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:12.383086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:12.383056    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.545751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:12.545711    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:12.700830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:12.700795    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:13.980146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:13.980116    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:15.871041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:15.871006    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:01:17.258378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:17.258346    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:17.423164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:17.423132    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:20.148676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:20.148619    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:20.161312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:20.161287    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:20.508668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:20.508628    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:01:21.835054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:21.835016    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.080361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:26.080324    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:26.427878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:26.427848    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:28.097989 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:01:28.097953    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:01:30.834805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:30.834772    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:01:31.013135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:31.013104    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:35.609597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:35.609565    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:40.230197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:40.230161    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.310790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.310756    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"]
Oct 30 22:01:41.314304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.314284    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.317717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.317697    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-dshm\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.317814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.317740    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-home\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.317854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.317832    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbltk\" (UniqueName: \"kubernetes.io/projected/eea4a889-5455-4f4a-b278-31c0d2d45d84-kube-api-access-gbltk\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.317887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.317878    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea4a889-5455-4f4a-b278-31c0d2d45d84-tls-certs\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.317921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.317897    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-tmp-dir\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.317977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.317963    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-model-cache\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.411435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.411398    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"kube-root-ca.crt\""
Oct 30 22:01:41.411435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.411424    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"openshift-service-ca.crt\""
Oct 30 22:01:41.411435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.411398    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"simulator-w1-e65019-kserve-self-signed-certs\""
Oct 30 22:01:41.411702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.411445    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e65019\"/\"default-dockercfg-dqqb4\""
Oct 30 22:01:41.413026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.413008    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"]
Oct 30 22:01:41.418254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418235    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-home\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418288    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbltk\" (UniqueName: \"kubernetes.io/projected/eea4a889-5455-4f4a-b278-31c0d2d45d84-kube-api-access-gbltk\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418313    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea4a889-5455-4f4a-b278-31c0d2d45d84-tls-certs\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418331    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-tmp-dir\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418350    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-model-cache\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418375    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-dshm\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418720    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-home\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418737    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-tmp-dir\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.418889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.418799    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-model-cache\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.420675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.420652    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-dshm\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.420925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.420909    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea4a889-5455-4f4a-b278-31c0d2d45d84-tls-certs\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.425767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.425727    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:41.516287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.516260    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbltk\" (UniqueName: \"kubernetes.io/projected/eea4a889-5455-4f4a-b278-31c0d2d45d84-kube-api-access-gbltk\") pod \"simulator-w1-e65019-kserve-68f88958d4-cq4nh\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") " pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.626267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.626178    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:41.995675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:41.995647    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"]
Oct 30 22:01:41.998101 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:01:41.998078    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeea4a889_5455_4f4a_b278_31c0d2d45d84.slice/crio-5bccc71754046d81801b8813b401ad49fe0f57e875ac221b34af858a87d0dc03 WatchSource:0}: Error finding container 5bccc71754046d81801b8813b401ad49fe0f57e875ac221b34af858a87d0dc03: Status 404 returned error can't find the container with id 5bccc71754046d81801b8813b401ad49fe0f57e875ac221b34af858a87d0dc03
Oct 30 22:01:42.606258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:42.606220    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" event={"ID":"eea4a889-5455-4f4a-b278-31c0d2d45d84","Type":"ContainerStarted","Data":"e2efe4261aa23125a83c8fffdefdf882080a60ac16c72beab0ec864a8179fa37"}
Oct 30 22:01:42.606258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:42.606265    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" event={"ID":"eea4a889-5455-4f4a-b278-31c0d2d45d84","Type":"ContainerStarted","Data":"5bccc71754046d81801b8813b401ad49fe0f57e875ac221b34af858a87d0dc03"}
Oct 30 22:01:42.626325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:42.626278    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:42.718987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:42.718902    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podStartSLOduration=1.718887353 podStartE2EDuration="1.718887353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:42.718457191 +0000 UTC m=+879.053033961" watchObservedRunningTime="2026-10-30 22:01:42.718887353 +0000 UTC m=+879.053464121"
Oct 30 22:01:44.771848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:44.771813    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:47.739509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:47.739476    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:01:49.342979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:49.342947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:50.433292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.433252    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"]
Oct 30 22:01:50.436801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.436777    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.497159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.497124    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-dshm\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.497340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.497221    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9114be59-0719-48bc-9695-933cf4606b43-tls-certs\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.497340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.497326    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-model-cache\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.497421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.497386    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-tmp-dir\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.497468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.497427    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-home\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.497526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.497502    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jq4\" (UniqueName: \"kubernetes.io/projected/9114be59-0719-48bc-9695-933cf4606b43-kube-api-access-r5jq4\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.532993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.532964    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b47e8d\"/\"premium-w3-b47e8d-kserve-self-signed-certs\""
Oct 30 22:01:50.534539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.534518    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"]
Oct 30 22:01:50.546594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.546569    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:50.598206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.598162    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-tmp-dir\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.598990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.598966    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-home\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.599483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.599452    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jq4\" (UniqueName: \"kubernetes.io/projected/9114be59-0719-48bc-9695-933cf4606b43-kube-api-access-r5jq4\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.600169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.600150    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-dshm\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.600725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.600704    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9114be59-0719-48bc-9695-933cf4606b43-tls-certs\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.601421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.601401    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-model-cache\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.601943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.598864    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-tmp-dir\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.602077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.601835    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-model-cache\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.602179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.599396    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-home\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.607223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.607199    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9114be59-0719-48bc-9695-933cf4606b43-tls-certs\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.609229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.609121    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-dshm\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.662660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.662615    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jq4\" (UniqueName: \"kubernetes.io/projected/9114be59-0719-48bc-9695-933cf4606b43-kube-api-access-r5jq4\") pod \"premium-w3-b47e8d-kserve-6c5959568f-hd8jr\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") " pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:50.699347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.699275    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:50.748802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:50.748760    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:51.129957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.129931    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"]
Oct 30 22:01:51.132375 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:01:51.132346    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9114be59_0719_48bc_9695_933cf4606b43.slice/crio-206b59aba099353c36f8f74229762a83817b6ffedb93eb085b863414772296be WatchSource:0}: Error finding container 206b59aba099353c36f8f74229762a83817b6ffedb93eb085b863414772296be: Status 404 returned error can't find the container with id 206b59aba099353c36f8f74229762a83817b6ffedb93eb085b863414772296be
Oct 30 22:01:51.626883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.626851    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:51.639851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.639818    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:51.651757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.651721    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" event={"ID":"9114be59-0719-48bc-9695-933cf4606b43","Type":"ContainerStarted","Data":"b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a"}
Oct 30 22:01:51.651908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.651765    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" event={"ID":"9114be59-0719-48bc-9695-933cf4606b43","Type":"ContainerStarted","Data":"206b59aba099353c36f8f74229762a83817b6ffedb93eb085b863414772296be"}
Oct 30 22:01:51.663220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.663193    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:01:51.749723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.749690    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:01:51.979425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:51.979359    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podStartSLOduration=1.9793384550000002 podStartE2EDuration="1.979338455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:51.977599617 +0000 UTC m=+888.312176386" watchObservedRunningTime="2026-10-30 22:01:51.979338455 +0000 UTC m=+888.313915227"
Oct 30 22:01:53.979278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:53.979242    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:56.737326 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:56.737290    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:01:58.096970 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:01:58.096935    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:01:58.620759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:01:58.620681    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:00.749961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:00.749933    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:02:00.762442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:00.762416    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:02:01.711541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:01.711517    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:02:03.241594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:03.241550    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:03.894893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:03.894864    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:02:03.898673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:03.898648    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:02:07.832673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:07.832613    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:12.382415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:12.382377    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:02:12.457668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:12.457611    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:14.327329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:14.327246    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:16.934785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:16.934752    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:21.449063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.449025    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:21.615151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.615117    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"]
Oct 30 22:02:21.618564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.618541    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.635327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.635303    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tls-certs\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.635458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.635337    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-model-cache\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.635458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.635367    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-home\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.635547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.635500    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tmp-dir\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.635589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.635550    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtjq\" (UniqueName: \"kubernetes.io/projected/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-kube-api-access-7gtjq\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.635664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.635623    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-dshm\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.668367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.668341    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"default-dockercfg-ljjzd\""
Oct 30 22:02:21.715405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.715314    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"openshift-service-ca.crt\""
Oct 30 22:02:21.715405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.715359    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"kube-root-ca.crt\""
Oct 30 22:02:21.717000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.716980    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"]
Oct 30 22:02:21.727893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.727858    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:21.737044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737024    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tls-certs\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737052    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-model-cache\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737078    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-home\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737101    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tmp-dir\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737124    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtjq\" (UniqueName: \"kubernetes.io/projected/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-kube-api-access-7gtjq\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737149    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-dshm\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737428    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-home\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737510    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tmp-dir\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.737575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.737530    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-model-cache\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.739390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.739370    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-dshm\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.773998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.773964    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e524ce\"/\"premium-w4-e524ce-kserve-self-signed-certs\""
Oct 30 22:02:21.784795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.784732    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tls-certs\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.795538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.795513    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtjq\" (UniqueName: \"kubernetes.io/projected/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-kube-api-access-7gtjq\") pod \"premium-w4-e524ce-kserve-569fb76f98-nvzsw\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") " pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:21.937777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:21.937740    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:22.308967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:22.308942    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"]
Oct 30 22:02:22.311312 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:02:22.311290    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddccedd2d_6a0b_4c71_bb7c_eda4063e6092.slice/crio-23ac3756e3c643723f6e33644af4cb8afe9c38ce379c559dbd7ccb46755db9c0 WatchSource:0}: Error finding container 23ac3756e3c643723f6e33644af4cb8afe9c38ce379c559dbd7ccb46755db9c0: Status 404 returned error can't find the container with id 23ac3756e3c643723f6e33644af4cb8afe9c38ce379c559dbd7ccb46755db9c0
Oct 30 22:02:22.808465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:22.808432    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" event={"ID":"dccedd2d-6a0b-4c71-bb7c-eda4063e6092","Type":"ContainerStarted","Data":"07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2"}
Oct 30 22:02:22.808465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:22.808472    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" event={"ID":"dccedd2d-6a0b-4c71-bb7c-eda4063e6092","Type":"ContainerStarted","Data":"23ac3756e3c643723f6e33644af4cb8afe9c38ce379c559dbd7ccb46755db9c0"}
Oct 30 22:02:22.920491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:22.920436    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podStartSLOduration=1.920421003 podStartE2EDuration="1.920421003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:22.91980077 +0000 UTC m=+919.254377540" watchObservedRunningTime="2026-10-30 22:02:22.920421003 +0000 UTC m=+919.254997780"
Oct 30 22:02:22.937979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:22.937946    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:26.117678 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:26.117616    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:28.098887 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:02:28.098851    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:02:30.664751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:30.664715    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:30.947692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:30.947585    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:02:31.938304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:31.938265    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:31.950863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:31.950835    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:32.868878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:32.868848    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:02:35.378362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:35.378322    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:40.012843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:40.012812    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:44.535649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:44.535611    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:45.886708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:45.886672    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:49.123178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:49.123145    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:53.741256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:53.741216    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:58.099282 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:02:58.099243    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:02:58.358305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:58.358227    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:02:59.357383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:02:59.357350    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:03:00.921475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:00.921440    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"]
Oct 30 22:03:00.924877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:00.924851    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:00.974713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:00.974683    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:03:00.974862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:00.974820    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:03:01.019031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.018996    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.019196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.019064    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.019196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.019165    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/d988eafd-fdaf-40a8-8466-97a2b28259c0-kube-api-access-d9xfh\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.021536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.021517    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:03:01.023301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.023282    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"]
Oct 30 22:03:01.074860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.074830    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:01.120733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.120697    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/d988eafd-fdaf-40a8-8466-97a2b28259c0-kube-api-access-d9xfh\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.120946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.120762    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.120946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.120807    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.121463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.121440    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.123155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.123135    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.179380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.179297    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/d988eafd-fdaf-40a8-8466-97a2b28259c0-kube-api-access-d9xfh\") pod \"maas-api-revoke-sub-70b90db6-ps6cz\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.237715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.237684    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:01.603498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:01.603473    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"]
Oct 30 22:03:01.605695 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:03:01.605667    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd988eafd_fdaf_40a8_8466_97a2b28259c0.slice/crio-ebe045e4889bf9eb07ed7973dfbd82da2a46c4a934f3c5d5556803b6e7fed9c8 WatchSource:0}: Error finding container ebe045e4889bf9eb07ed7973dfbd82da2a46c4a934f3c5d5556803b6e7fed9c8: Status 404 returned error can't find the container with id ebe045e4889bf9eb07ed7973dfbd82da2a46c4a934f3c5d5556803b6e7fed9c8
Oct 30 22:03:02.023811 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:02.023723    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz" event={"ID":"d988eafd-fdaf-40a8-8466-97a2b28259c0","Type":"ContainerStarted","Data":"ebe045e4889bf9eb07ed7973dfbd82da2a46c4a934f3c5d5556803b6e7fed9c8"}
Oct 30 22:03:03.025411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:03.025381    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:03.030315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:03.030291    5503 generic.go:356] "Generic (PLEG): container finished" podID="d988eafd-fdaf-40a8-8466-97a2b28259c0" containerID="014ccc8a7381cde8ec32b09f781f0d1f54dca896bd512a51f24ad34b68579459" exitCode=0
Oct 30 22:03:03.030446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:03.030377    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz" event={"ID":"d988eafd-fdaf-40a8-8466-97a2b28259c0","Type":"ContainerDied","Data":"014ccc8a7381cde8ec32b09f781f0d1f54dca896bd512a51f24ad34b68579459"}
Oct 30 22:03:04.171392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.171368    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:04.350416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.350329    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-xks-ca\") pod \"d988eafd-fdaf-40a8-8466-97a2b28259c0\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") "
Oct 30 22:03:04.350416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.350391    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-openshift-ca\") pod \"d988eafd-fdaf-40a8-8466-97a2b28259c0\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") "
Oct 30 22:03:04.350667 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.350422    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d988eafd-fdaf-40a8-8466-97a2b28259c0-kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/d988eafd-fdaf-40a8-8466-97a2b28259c0-kube-api-access-d9xfh\") pod \"d988eafd-fdaf-40a8-8466-97a2b28259c0\" (UID: \"d988eafd-fdaf-40a8-8466-97a2b28259c0\") "
Oct 30 22:03:04.350818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.350789    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-openshift-ca" pod "d988eafd-fdaf-40a8-8466-97a2b28259c0" (UID: "d988eafd-fdaf-40a8-8466-97a2b28259c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:04.352382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.352359    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-xks-ca" pod "d988eafd-fdaf-40a8-8466-97a2b28259c0" (UID: "d988eafd-fdaf-40a8-8466-97a2b28259c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:04.352481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.352469    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d988eafd-fdaf-40a8-8466-97a2b28259c0-kube-api-access-d9xfh" pod "d988eafd-fdaf-40a8-8466-97a2b28259c0" (UID: "d988eafd-fdaf-40a8-8466-97a2b28259c0"). InnerVolumeSpecName "kube-api-access-d9xfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:04.451466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.451436    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:04.451466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.451467    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d988eafd-fdaf-40a8-8466-97a2b28259c0-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:04.451660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:04.451484    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xfh\" (UniqueName: \"kubernetes.io/projected/d988eafd-fdaf-40a8-8466-97a2b28259c0-kube-api-access-d9xfh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:05.041130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:05.041091    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz" event={"ID":"d988eafd-fdaf-40a8-8466-97a2b28259c0","Type":"ContainerDied","Data":"ebe045e4889bf9eb07ed7973dfbd82da2a46c4a934f3c5d5556803b6e7fed9c8"}
Oct 30 22:03:05.041306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:05.041146    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebe045e4889bf9eb07ed7973dfbd82da2a46c4a934f3c5d5556803b6e7fed9c8"
Oct 30 22:03:05.041306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:05.041104    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"
Oct 30 22:03:07.631026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:07.630992    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:12.146058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:12.146020    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:13.574292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:13.574258    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:14.644775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:14.644745    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:16.814923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:16.814888    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:21.439762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:21.439729    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:26.024596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:26.024566    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:27.623644 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:27.623612    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:03:28.101996 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:03:28.101959    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:03:30.708625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:30.708589    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:30.880040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:30.880001    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:31.041504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:31.041419    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:35.200716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:35.200686    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:35.362422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:35.362390    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:39.903551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:39.903517    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:44.725798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:44.725746    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:46.655589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.655557    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"]
Oct 30 22:03:46.656083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.656066    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d988eafd-fdaf-40a8-8466-97a2b28259c0" containerName="revoke-keys"
Oct 30 22:03:46.656126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.656084    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d988eafd-fdaf-40a8-8466-97a2b28259c0" containerName="revoke-keys"
Oct 30 22:03:46.656173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.656163    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d988eafd-fdaf-40a8-8466-97a2b28259c0" containerName="revoke-keys"
Oct 30 22:03:46.658459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.658439    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.677925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.677897    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.677932    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.677951    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.678085    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.678138    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.678217    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.678259    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.678277    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264zf\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-kube-api-access-264zf\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.678380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.678306    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.708103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.708081    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b47e8d-openshift-default-dockercfg-bxx6t\""
Oct 30 22:03:46.757003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.756978    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"]
Oct 30 22:03:46.779919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.779894    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.779938    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.779976    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.779995    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780011    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264zf\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-kube-api-access-264zf\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780193    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780252    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780288    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780315    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780348    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780528    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780810    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.780969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.780945    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.781135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.781109    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.782424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.782395    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.782906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.782885    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.838331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.838297    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.838480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.838437    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264zf\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-kube-api-access-264zf\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:46.970356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:46.970281    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:47.385129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:47.385098    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"]
Oct 30 22:03:47.387055 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:03:47.387031    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafd3c91_fc3b_4ab3_8e1e_6f6ee23a2aa1.slice/crio-25eb99d6f64ed97e34f41284cd1c1e2f8c4fa915a9d94a162916b86119d05683 WatchSource:0}: Error finding container 25eb99d6f64ed97e34f41284cd1c1e2f8c4fa915a9d94a162916b86119d05683: Status 404 returned error can't find the container with id 25eb99d6f64ed97e34f41284cd1c1e2f8c4fa915a9d94a162916b86119d05683
Oct 30 22:03:47.389443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:47.389409    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:47.389514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:47.389474    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:47.389556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:47.389512    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:48.261315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:48.261278    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" event={"ID":"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1","Type":"ContainerStarted","Data":"44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5"}
Oct 30 22:03:48.261808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:48.261327    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" event={"ID":"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1","Type":"ContainerStarted","Data":"25eb99d6f64ed97e34f41284cd1c1e2f8c4fa915a9d94a162916b86119d05683"}
Oct 30 22:03:48.432660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:48.430704    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podStartSLOduration=2.430684357 podStartE2EDuration="2.430684357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:48.429169713 +0000 UTC m=+1004.763746507" watchObservedRunningTime="2026-10-30 22:03:48.430684357 +0000 UTC m=+1004.765261126"
Oct 30 22:03:48.971445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:48.971406    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:48.972941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:48.972911    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:03:48.973069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:48.972964    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:03:49.340418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:49.340328    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:49.971629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:49.971593    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:03:49.971827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:49.971707    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:03:50.970805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:50.970774    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:03:50.971262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:50.971235    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:03:51.971152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:51.971112    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:03:51.971663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:51.971172    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:03:52.971373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:52.971326    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:03:52.971792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:52.971393    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:03:53.655107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.655054    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:53.806270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.806231    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"]
Oct 30 22:03:53.810519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.810494    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.861060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.861025    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7f66ab-openshift-default-dockercfg-pk2pq\""
Oct 30 22:03:53.876778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.876733    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.876981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.876791    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tww\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-kube-api-access-t2tww\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.876981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.876832    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.876981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.876862    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c34e5e-9c39-46b2-808f-83192790e90a-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.876981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.876926    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.876981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.876973    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.877305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.877013    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c34e5e-9c39-46b2-808f-83192790e90a-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.877305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.877072    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.877305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.877104    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.920007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.919912    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"]
Oct 30 22:03:53.935504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.935470    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:53.989904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.989862    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.992520    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.992813    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.992874    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tww\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-kube-api-access-t2tww\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.992990    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.993071    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c34e5e-9c39-46b2-808f-83192790e90a-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.993178    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.993254    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.993307    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c34e5e-9c39-46b2-808f-83192790e90a-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.990856    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.995146    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:53.999670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:53.996617    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.010688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.010590    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.011673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.011630    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c34e5e-9c39-46b2-808f-83192790e90a-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.022338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.022264    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c34e5e-9c39-46b2-808f-83192790e90a-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.033599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.033565    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.053630 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.053598    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.053915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.053890    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tww\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-kube-api-access-t2tww\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.125942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.125903    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:54.139146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.138827    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:54.139872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.139787    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:54.141161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.141141    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:03:54.561501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.561469    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"]
Oct 30 22:03:54.563443 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:03:54.563414    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c34e5e_9c39_46b2_808f_83192790e90a.slice/crio-4c4ecead0676e865be5be7d2582daaf13ee10943d2ced5781fe0e69471077b7d WatchSource:0}: Error finding container 4c4ecead0676e865be5be7d2582daaf13ee10943d2ced5781fe0e69471077b7d: Status 404 returned error can't find the container with id 4c4ecead0676e865be5be7d2582daaf13ee10943d2ced5781fe0e69471077b7d
Oct 30 22:03:54.565695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.565663    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:54.565757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.565728    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:54.565808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:54.565765    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:55.297386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:55.297352    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" event={"ID":"13c34e5e-9c39-46b2-808f-83192790e90a","Type":"ContainerStarted","Data":"4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061"}
Oct 30 22:03:55.297875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:55.297413    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" event={"ID":"13c34e5e-9c39-46b2-808f-83192790e90a","Type":"ContainerStarted","Data":"4c4ecead0676e865be5be7d2582daaf13ee10943d2ced5781fe0e69471077b7d"}
Oct 30 22:03:55.412970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:55.412917    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podStartSLOduration=2.412901717 podStartE2EDuration="2.412901717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:55.411561143 +0000 UTC m=+1011.746137914" watchObservedRunningTime="2026-10-30 22:03:55.412901717 +0000 UTC m=+1011.747478546"
Oct 30 22:03:56.127660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:56.126961    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:03:56.135666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:56.132742    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:03:56.135666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:56.132797    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 22:03:57.131075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:57.131034    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.131694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:57.131663    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 22:03:58.097770 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:03:58.097733    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:03:58.127486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.127019    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.127486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.127093    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 22:03:58.181710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.181676    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"]
Oct 30 22:03:58.184786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.184767    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349387    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349451    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349572    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349601    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349717    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349772    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv8n7\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-kube-api-access-qv8n7\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349828 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349813    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349854    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.349902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.349890    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.403373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.403341    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"]
Oct 30 22:03:58.450730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.450688    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.450925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.450769    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.450925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.450801    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.450925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.450848    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.450925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.450879    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv8n7\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-kube-api-access-qv8n7\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.450939    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.450990    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.451033    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.451098    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.451207    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.451468    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.451733    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.451923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.451781    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.452065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.451797    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.454187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.454157    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.454478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.454460    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.511538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.511509    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv8n7\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-kube-api-access-qv8n7\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.511705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.511511    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-zjzp6\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:58.677824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.677790    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:03:58.797998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:58.797967    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:03:59.137711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.136711    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.137711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.136788    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 22:03:59.246198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.246042    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"]
Oct 30 22:03:59.249765 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:03:59.249737    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd4432e_5f73_4ef7_9e6f_2a947b0766e7.slice/crio-ebf9b3a0d693821400cb90dd85eb87027188dd746a69c42d2c2ecd7276615df5 WatchSource:0}: Error finding container ebf9b3a0d693821400cb90dd85eb87027188dd746a69c42d2c2ecd7276615df5: Status 404 returned error can't find the container with id ebf9b3a0d693821400cb90dd85eb87027188dd746a69c42d2c2ecd7276615df5
Oct 30 22:03:59.252522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.252484    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:59.252591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.252575    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:59.252653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.252613    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:03:59.340664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.340183    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" event={"ID":"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7","Type":"ContainerStarted","Data":"2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986"}
Oct 30 22:03:59.340664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.340252    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" event={"ID":"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7","Type":"ContainerStarted","Data":"ebf9b3a0d693821400cb90dd85eb87027188dd746a69c42d2c2ecd7276615df5"}
Oct 30 22:03:59.459924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.459863    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podStartSLOduration=1.459847514 podStartE2EDuration="1.459847514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:59.456263227 +0000 UTC m=+1015.790839996" watchObservedRunningTime="2026-10-30 22:03:59.459847514 +0000 UTC m=+1015.794424282"
Oct 30 22:03:59.798955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:03:59.798874    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:04:00.129200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:00.129093    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.130321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:00.130288    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Oct 30 22:04:00.610841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:00.610799    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:04:00.800357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:00.800322    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:04:00.800513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:00.800368    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 22:04:01.798530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:01.798483    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:04:01.798530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:01.798534    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 22:04:02.168891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:02.168837    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:02.169110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:02.168932    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:02.702928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:02.702896    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:04:02.703535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:02.703516    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:04:02.704024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:02.704005    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:04:02.798911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:02.798876    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:04:02.799326 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:02.798931    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 22:04:03.731222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:03.731183    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:03.798553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:03.798515    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:04:03.798877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:03.798836    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 22:04:04.798520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:04.798486    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:04:04.798930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:04.798539    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 22:04:05.798870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:05.798837    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:04:05.799262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:05.798888    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 22:04:06.799461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:06.799423    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Oct 30 22:04:06.799888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:06.799508    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Oct 30 22:04:08.439018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:08.438985    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:08.799444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:08.799339    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:08.799444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:08.799429    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:09.840786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:09.840744    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:09.841308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:09.840795    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:10.301070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:10.301039    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:04:10.301473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:10.301446    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:04:10.302302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:10.302282    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:04:13.055274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.055228    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:13.280961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.280927    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"]
Oct 30 22:04:13.285664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.285627    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.332754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.332681    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.332754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.332743    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f9080d-92e8-459f-a858-261e5bb09d3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.332903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.332830    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.332903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.332865    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.332985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.332953    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.333038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.332989    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.333112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.333098    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.333156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.333122    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8xg\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-kube-api-access-6n8xg\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.333197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.333177    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.384831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.384799    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"]
Oct 30 22:04:13.434562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434533    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.434562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434566    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8xg\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-kube-api-access-6n8xg\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.434804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434625    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.434804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434665    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.434804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434713    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f9080d-92e8-459f-a858-261e5bb09d3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.434804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434745    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.434804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434770    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.435052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434812    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.435052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.434848    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.435052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.435013    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.435052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.435031    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.435261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.435214    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.435261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.435228    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.435806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.435778    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f9080d-92e8-459f-a858-261e5bb09d3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.437406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.437374    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.437508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.437487    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.493402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.493373    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.493574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.493485    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8xg\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-kube-api-access-6n8xg\") pod \"maas-default-gateway-openshift-default-6f54779779-58nlg\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:13.598353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:13.598264    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:14.025220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.025169    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"]
Oct 30 22:04:14.028748 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:04:14.028715    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f9080d_92e8_459f_a858_261e5bb09d3d.slice/crio-0fb23225dcc95edece0eddbb4ba8565f923a414a7ad1f6b6126bab8bd0a7926d WatchSource:0}: Error finding container 0fb23225dcc95edece0eddbb4ba8565f923a414a7ad1f6b6126bab8bd0a7926d: Status 404 returned error can't find the container with id 0fb23225dcc95edece0eddbb4ba8565f923a414a7ad1f6b6126bab8bd0a7926d
Oct 30 22:04:14.031211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.031173    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:04:14.031320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.031277    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:04:14.031320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.031313    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:04:14.446455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.446418    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" event={"ID":"c4f9080d-92e8-459f-a858-261e5bb09d3d","Type":"ContainerStarted","Data":"7905855a62daa084e08caad08106819f680853f127a10895db081e03ec1dc99e"}
Oct 30 22:04:14.446885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.446484    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" event={"ID":"c4f9080d-92e8-459f-a858-261e5bb09d3d","Type":"ContainerStarted","Data":"0fb23225dcc95edece0eddbb4ba8565f923a414a7ad1f6b6126bab8bd0a7926d"}
Oct 30 22:04:14.599143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.599104    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:14.611967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:14.611910    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podStartSLOduration=1.61189503 podStartE2EDuration="1.61189503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:14.610139932 +0000 UTC m=+1030.944716702" watchObservedRunningTime="2026-10-30 22:04:14.61189503 +0000 UTC m=+1030.946471798"
Oct 30 22:04:15.004292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:15.004260    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:15.600195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:15.600158    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.600574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:15.600213    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:16.598808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:16.598767    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.599057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:16.598824    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:17.599201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:17.599165    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.599618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:17.599231    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:17.778209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:17.778169    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:18.599518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:18.599476    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.599959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:18.599540    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:19.599275 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:19.599231    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.599600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:19.599294    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:20.598582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:20.598546    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.598793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:20.598607    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:21.599219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:21.599178    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.599655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:21.599262    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 22:04:22.626441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:22.626406    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:23.600288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:23.600241    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:23.600497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:23.600327    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:24.641844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:24.641793    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-58nlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:24.642270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:24.641876    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:24.963839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:24.963806    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:24.964155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:24.964131    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:24.965113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:24.965091    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:04:27.238651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:27.238601    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:28.099101 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:04:28.099065    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:04:31.809152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:31.809120    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:36.414292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:36.414252    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:37.618340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:37.618302    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:41.011475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:41.011438    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:43.822835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:43.822799    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:45.526767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:45.526734    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:46.088248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:46.088214    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:04:46.845010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:46.844978    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:46.998274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:46.998242    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:50.138353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:50.138315    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:54.932848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:54.932808    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:04:58.100327 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:04:58.100289    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:04:59.692054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:04:59.689384    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:03.909156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:03.909121    5503 scope.go:122] "RemoveContainer" containerID="091b6cae3b4589023e700a87d6ac8da7cb5147d11b7cbc1807b847fc2b50415f"
Oct 30 22:05:04.338447 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:04.338367    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:07.914734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:07.914701    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:05:08.549344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.549310    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"]
Oct 30 22:05:08.555080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.555035    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.567676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.567631    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692a0bc0-19b2-42a2-b781-499bf4a62eef-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.567801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.567691    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.567801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.567765    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.567801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.567788    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.567928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.567812    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.567928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.567907    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.568048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.567939    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.568118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.568042    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.568118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.568091    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jbr\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-kube-api-access-n4jbr\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.661745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.661712    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"]
Oct 30 22:05:08.669707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669677    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.669887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669712    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.669887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669739    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.669887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669763    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.669887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669790    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.669887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669839    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.669887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669875    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jbr\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-kube-api-access-n4jbr\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.670218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.669994    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692a0bc0-19b2-42a2-b781-499bf4a62eef-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.670218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.670026    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.670218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.670144    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.670557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.670531    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.670692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.670562    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.670879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.670849    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.671184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.671156    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692a0bc0-19b2-42a2-b781-499bf4a62eef-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.672725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.672704    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.672995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.672975    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.734710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.734671    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.734877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.734821    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jbr\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-kube-api-access-n4jbr\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.870098 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.870015    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:08.925649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:08.925607    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:09.292210 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:05:09.292177    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692a0bc0_19b2_42a2_b781_499bf4a62eef.slice/crio-2e8554b06afa4b78f9553e548f1ee566448bb266d9a708028c4079c395ce6e37 WatchSource:0}: Error finding container 2e8554b06afa4b78f9553e548f1ee566448bb266d9a708028c4079c395ce6e37: Status 404 returned error can't find the container with id 2e8554b06afa4b78f9553e548f1ee566448bb266d9a708028c4079c395ce6e37
Oct 30 22:05:09.294689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:09.294654    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:05:09.294773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:09.294727    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:05:09.294773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:09.294756    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:05:09.294897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:09.294877    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"]
Oct 30 22:05:09.779404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:09.779363    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" event={"ID":"692a0bc0-19b2-42a2-b781-499bf4a62eef","Type":"ContainerStarted","Data":"9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa"}
Oct 30 22:05:09.779725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:09.779699    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" event={"ID":"692a0bc0-19b2-42a2-b781-499bf4a62eef","Type":"ContainerStarted","Data":"2e8554b06afa4b78f9553e548f1ee566448bb266d9a708028c4079c395ce6e37"}
Oct 30 22:05:09.870292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:09.870264    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:10.872527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:10.872493    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:05:10.872952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:10.872545    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:05:11.760430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.760377    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podStartSLOduration=3.7603569390000002 podStartE2EDuration="3.760356939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:09.922964165 +0000 UTC m=+1086.257540945" watchObservedRunningTime="2026-10-30 22:05:11.760356939 +0000 UTC m=+1088.094933709"
Oct 30 22:05:11.760878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.760860    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"]
Oct 30 22:05:11.764472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.764455    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.839333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.839292    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de44df9-846c-410e-aa8f-e456452b69ac-tls-certs\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.839674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.839652    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-home\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.843752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.843719    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-tmp-dir\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.843958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.843926    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-dshm\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.844239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.844213    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswhn\" (UniqueName: \"kubernetes.io/projected/6de44df9-846c-410e-aa8f-e456452b69ac-kube-api-access-dswhn\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.844320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.844277    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-model-cache\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.868933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.868901    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"e2e-conflict-b-d3c7fa24-kserve-self-signed-certs\""
Oct 30 22:05:11.870336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.870315    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:05:11.870440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.870371    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:05:11.871694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.871668    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:11.888729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.888706    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"]
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.958416    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswhn\" (UniqueName: \"kubernetes.io/projected/6de44df9-846c-410e-aa8f-e456452b69ac-kube-api-access-dswhn\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.958475    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-model-cache\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.958558    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de44df9-846c-410e-aa8f-e456452b69ac-tls-certs\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.958595    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-home\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.958620    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-tmp-dir\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.958667    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-dshm\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.960043    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-tmp-dir\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.960292    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-model-cache\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.960367    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-home\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.963661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.962915    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-dshm\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:11.970303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:11.970282    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de44df9-846c-410e-aa8f-e456452b69ac-tls-certs\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:12.047452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.047376    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswhn\" (UniqueName: \"kubernetes.io/projected/6de44df9-846c-410e-aa8f-e456452b69ac-kube-api-access-dswhn\") pod \"e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") " pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:12.076254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.076216    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:12.510195 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:05:12.510162    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6de44df9_846c_410e_aa8f_e456452b69ac.slice/crio-9a132abae9aaa100362b4f57b32e18ec20e36479ea9f0bf9bd7bf05716887511 WatchSource:0}: Error finding container 9a132abae9aaa100362b4f57b32e18ec20e36479ea9f0bf9bd7bf05716887511: Status 404 returned error can't find the container with id 9a132abae9aaa100362b4f57b32e18ec20e36479ea9f0bf9bd7bf05716887511
Oct 30 22:05:12.512174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.512148    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"]
Oct 30 22:05:12.798558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.798521    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6" event={"ID":"6de44df9-846c-410e-aa8f-e456452b69ac","Type":"ContainerStarted","Data":"c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016"}
Oct 30 22:05:12.798558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.798564    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6" event={"ID":"6de44df9-846c-410e-aa8f-e456452b69ac","Type":"ContainerStarted","Data":"9a132abae9aaa100362b4f57b32e18ec20e36479ea9f0bf9bd7bf05716887511"}
Oct 30 22:05:12.870882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.870782    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:05:12.870882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.870837    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:05:12.933844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:12.933780    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6" podStartSLOduration=1.9337597149999999 podStartE2EDuration="1.933759715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:12.931098621 +0000 UTC m=+1089.265675394" watchObservedRunningTime="2026-10-30 22:05:12.933759715 +0000 UTC m=+1089.268336484"
Oct 30 22:05:13.076954 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:13.076915    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:13.586955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:13.586902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:13.871379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:13.871300    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:05:13.871379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:13.871361    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:05:14.870855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:14.870820    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:05:14.871238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:14.870877    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:05:15.438781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:15.438741    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:15.870665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:15.870558    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:05:15.870816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:15.870652    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:05:16.875303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:16.875275    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:16.875745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:16.875733    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:16.876531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:16.876509    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:05:18.336082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:18.336047    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:22.076894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:22.076856    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:22.089687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:22.089655    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:22.893271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:22.893232    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:05:23.261738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:23.261697    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:28.097941 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:05:28.097906    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:05:28.228125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:28.228086    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:32.825001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:32.824960    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:37.529565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:37.529530    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:39.080858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.080825    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:05:39.664507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.664478    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"]
Oct 30 22:05:39.667914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.667898    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.791317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.791283    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"]
Oct 30 22:05:39.819525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.819502    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.819697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.819533    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.819697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.819561    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.819822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.819715    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.819822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.819744    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.819921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.819826    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.819988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.819970    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.820045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.820004    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.820099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.820048    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pbq\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-kube-api-access-p2pbq\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.921618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.921529    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.921618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.921571    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.921618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.921608    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pbq\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-kube-api-access-p2pbq\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.921937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.921721    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.921937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.921747    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.921937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.921773    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.921937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.921836    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.922131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.922015    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.922131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.922052    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.922131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.922077    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.922286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.922182    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.922386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.922361    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.922439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.922410    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.922439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.922413    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.924159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.924130    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.924348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.924332    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.984651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.984611    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pbq\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-kube-api-access-p2pbq\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:39.985359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:39.985335    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:40.281732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:40.281653    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:40.725468 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:05:40.725432    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24d013e_a975_46ed_9e2e_9109a7bfaea1.slice/crio-b3034a277c823c6dd2b8fdb537f4023f08c8f49da6c4217a04455ab9fd0e0a12 WatchSource:0}: Error finding container b3034a277c823c6dd2b8fdb537f4023f08c8f49da6c4217a04455ab9fd0e0a12: Status 404 returned error can't find the container with id b3034a277c823c6dd2b8fdb537f4023f08c8f49da6c4217a04455ab9fd0e0a12
Oct 30 22:05:40.727976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:40.727952    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"]
Oct 30 22:05:40.728359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:40.728329    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:05:40.728413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:40.728388    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:05:40.728458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:40.728416    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:05:40.979158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:40.979041    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" event={"ID":"d24d013e-a975-46ed-9e2e-9109a7bfaea1","Type":"ContainerStarted","Data":"37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4"}
Oct 30 22:05:40.979158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:40.979103    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" event={"ID":"d24d013e-a975-46ed-9e2e-9109a7bfaea1","Type":"ContainerStarted","Data":"b3034a277c823c6dd2b8fdb537f4023f08c8f49da6c4217a04455ab9fd0e0a12"}
Oct 30 22:05:41.101673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:41.101606    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podStartSLOduration=2.101591511 podStartE2EDuration="2.101591511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:41.101136516 +0000 UTC m=+1117.435713284" watchObservedRunningTime="2026-10-30 22:05:41.101591511 +0000 UTC m=+1117.436168280"
Oct 30 22:05:41.282108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:41.281975    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:41.283652 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:41.283611    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:05:41.283778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:41.283673    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:05:42.048203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:42.048169    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:42.283015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:42.282981    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:05:42.283427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:42.283038    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:05:43.282897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:43.282857    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:05:43.283060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:43.282940    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:05:43.759492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:43.759460    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:44.281990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:44.281943    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:05:44.282185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:44.282009    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:05:45.282884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:45.282845    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:05:45.283301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:45.282908    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:05:46.282557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:46.282512    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:05:46.282753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:46.282599    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:05:46.657816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:46.657763    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:47.282692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:47.282620    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:05:47.282885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:47.282763    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:05:48.287905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:48.287871    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:48.288373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:48.288314    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:48.289132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:48.289111    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:05:51.334312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:51.334275    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:54.670274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:54.670230    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:56.138087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:05:56.138048    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:05:58.098800 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:05:58.098764    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:06:00.720031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:00.719992    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:03.929780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:03.929745    5503 scope.go:122] "RemoveContainer" containerID="39c12999be69fdca65b9fcb140964c63895e74fc41e69b1abc5e27a485afbf94"
Oct 30 22:06:03.938215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:03.938188    5503 scope.go:122] "RemoveContainer" containerID="dff25c5dbbd7170ddc1526956a91e743ac1b10b0c15560bf7ef1b1378751b425"
Oct 30 22:06:05.238244 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:05.238213    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:09.047367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.047331    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"]
Oct 30 22:06:09.052480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.052445    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118393    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118443    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118576    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118673    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118724    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118751    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhgs\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-kube-api-access-lxhgs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118823    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118875    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.118925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.118912    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.219776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.219734    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.219946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.219847    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.219946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.219885    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.219945    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.219983    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.220041    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.220078    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhgs\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-kube-api-access-lxhgs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.220116    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.220149    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.220662    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.220873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.220845    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.224836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.223161    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.224836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.224010    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.224836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.224323    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.224836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.224520    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.225893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.225224    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.264136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.264098    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"]
Oct 30 22:06:09.282708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.282671    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhgs\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-kube-api-access-lxhgs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.283355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.283327    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.366210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.366143    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:09.639437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.639363    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"]
Oct 30 22:06:09.644242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.644215    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.699528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.699493    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d09fff\"/\"distinct-w0-d09fff-kserve-self-signed-certs\""
Oct 30 22:06:09.701463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.701433    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"]
Oct 30 22:06:09.727405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.727365    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b406a09b-7afd-40cb-b013-25ca20c573df-tls-certs\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.727546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.727521    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-model-cache\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.727602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.727553    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8pc\" (UniqueName: \"kubernetes.io/projected/b406a09b-7afd-40cb-b013-25ca20c573df-kube-api-access-5s8pc\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.734014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.727669    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-dshm\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.734014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.727724    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-home\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.734014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.727777    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-tmp-dir\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.791209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.791173    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:09.799136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.799012    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"]
Oct 30 22:06:09.799859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.799824    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:06:09.799950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.799888    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:06:09.799950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.799932    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:06:09.828386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.828359    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-dshm\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.828503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.828465    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-home\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.828503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.828494    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-tmp-dir\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.828627 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.828554    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b406a09b-7afd-40cb-b013-25ca20c573df-tls-certs\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.828700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.828689    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-model-cache\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.828762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.828718    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8pc\" (UniqueName: \"kubernetes.io/projected/b406a09b-7afd-40cb-b013-25ca20c573df-kube-api-access-5s8pc\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.828867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.828843    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-home\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.829188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.829163    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-model-cache\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.829188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.829184    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-tmp-dir\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.831134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.831115    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-dshm\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.831285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.831267    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b406a09b-7afd-40cb-b013-25ca20c573df-tls-certs\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.898077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.897994    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8pc\" (UniqueName: \"kubernetes.io/projected/b406a09b-7afd-40cb-b013-25ca20c573df-kube-api-access-5s8pc\") pod \"distinct-w0-d09fff-kserve-b89f5cc66-d2fnc\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") " pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:09.957938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:09.957888    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:10.173534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.171192    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" event={"ID":"5845e6a9-82a9-4bee-86bf-ca8637bf48b7","Type":"ContainerStarted","Data":"b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0"}
Oct 30 22:06:10.173534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.171265    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" event={"ID":"5845e6a9-82a9-4bee-86bf-ca8637bf48b7","Type":"ContainerStarted","Data":"a738e15b1bcfb52c8f08e65c58b99ba8026c05ea7646491a1e9c3376c88aa7f9"}
Oct 30 22:06:10.308311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.308257    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podStartSLOduration=2.3082402220000002 podStartE2EDuration="2.308240222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:10.307727518 +0000 UTC m=+1146.642304287" watchObservedRunningTime="2026-10-30 22:06:10.308240222 +0000 UTC m=+1146.642817027"
Oct 30 22:06:10.367253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.367216    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:10.368895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.368867    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:10.369013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.368926    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:06:10.433889 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:06:10.433848    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb406a09b_7afd_40cb_b013_25ca20c573df.slice/crio-64d1c3b81cccc9916507a7b687fd2574bc05a8881f9b35a0c7fd92a813b5dc09 WatchSource:0}: Error finding container 64d1c3b81cccc9916507a7b687fd2574bc05a8881f9b35a0c7fd92a813b5dc09: Status 404 returned error can't find the container with id 64d1c3b81cccc9916507a7b687fd2574bc05a8881f9b35a0c7fd92a813b5dc09
Oct 30 22:06:10.435185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.435159    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"]
Oct 30 22:06:10.985487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:10.985452    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:06:11.178332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:11.178300    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" event={"ID":"b406a09b-7afd-40cb-b013-25ca20c573df","Type":"ContainerStarted","Data":"853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48"}
Oct 30 22:06:11.178761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:11.178343    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" event={"ID":"b406a09b-7afd-40cb-b013-25ca20c573df","Type":"ContainerStarted","Data":"64d1c3b81cccc9916507a7b687fd2574bc05a8881f9b35a0c7fd92a813b5dc09"}
Oct 30 22:06:11.202630 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:11.202601    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:11.329613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:11.329494    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podStartSLOduration=2.329476091 podStartE2EDuration="2.329476091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:11.329051884 +0000 UTC m=+1147.663628654" watchObservedRunningTime="2026-10-30 22:06:11.329476091 +0000 UTC m=+1147.664052861"
Oct 30 22:06:11.366680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:11.366611    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:11.366843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:11.366706    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:06:11.958712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:11.958675    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:12.366911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:12.366818    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:12.366911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:12.366872    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:06:13.366465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:13.366428    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:13.366673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:13.366490    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:06:14.366897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:14.366856    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:14.367368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:14.366914    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:06:14.803078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:14.803035    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:15.367019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:15.366987    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:15.367403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:15.367051    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:06:15.756735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:15.756695    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:16.367350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:16.367315    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:06:16.367776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:16.367368    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:06:17.372130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:17.372098    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:17.372548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:17.372525    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:17.373168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:17.373150    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:06:19.622680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:19.621735    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:19.958050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:19.958015    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:19.972499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:19.972474    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:20.240454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:20.240370    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:06:22.966871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:22.966831    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"]
Oct 30 22:06:23.028790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:23.028759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:24.288039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:24.288007    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:28.099875 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:06:28.099831    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:06:28.462532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:28.462492    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:28.988262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:28.988198    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:33.548474 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:33.548438    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:34.374961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:34.374921    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:35.814053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:35.814018    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:06:37.987754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:37.987719    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6" podUID="6de44df9-846c-410e-aa8f-e456452b69ac" containerName="main" containerID="cri-o://c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016" gracePeriod=45
Oct 30 22:06:38.258323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.258301    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:06:38.310746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.310717    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-model-cache\") pod \"6de44df9-846c-410e-aa8f-e456452b69ac\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") "
Oct 30 22:06:38.310873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.310826    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-tmp-dir\") pod \"6de44df9-846c-410e-aa8f-e456452b69ac\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") "
Oct 30 22:06:38.310873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.310845    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-dshm\") pod \"6de44df9-846c-410e-aa8f-e456452b69ac\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") "
Oct 30 22:06:38.310873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.310864    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de44df9-846c-410e-aa8f-e456452b69ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/6de44df9-846c-410e-aa8f-e456452b69ac-tls-certs\") pod \"6de44df9-846c-410e-aa8f-e456452b69ac\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") "
Oct 30 22:06:38.311107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.310889    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-home\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-home\") pod \"6de44df9-846c-410e-aa8f-e456452b69ac\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") "
Oct 30 22:06:38.311107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.310924    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de44df9-846c-410e-aa8f-e456452b69ac-kube-api-access-dswhn\" (UniqueName: \"kubernetes.io/projected/6de44df9-846c-410e-aa8f-e456452b69ac-kube-api-access-dswhn\") pod \"6de44df9-846c-410e-aa8f-e456452b69ac\" (UID: \"6de44df9-846c-410e-aa8f-e456452b69ac\") "
Oct 30 22:06:38.311107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.310991    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-model-cache" pod "6de44df9-846c-410e-aa8f-e456452b69ac" (UID: "6de44df9-846c-410e-aa8f-e456452b69ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:38.311262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.311146    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-home" pod "6de44df9-846c-410e-aa8f-e456452b69ac" (UID: "6de44df9-846c-410e-aa8f-e456452b69ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:38.311262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.311162    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-tmp-dir" pod "6de44df9-846c-410e-aa8f-e456452b69ac" (UID: "6de44df9-846c-410e-aa8f-e456452b69ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:38.311452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.311432    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:38.311521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.311454    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:38.311521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.311463    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:38.313141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.313117    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de44df9-846c-410e-aa8f-e456452b69ac-kube-api-access-dswhn" pod "6de44df9-846c-410e-aa8f-e456452b69ac" (UID: "6de44df9-846c-410e-aa8f-e456452b69ac"). InnerVolumeSpecName "kube-api-access-dswhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:38.313250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.313156    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-dshm" pod "6de44df9-846c-410e-aa8f-e456452b69ac" (UID: "6de44df9-846c-410e-aa8f-e456452b69ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:38.313250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.313159    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de44df9-846c-410e-aa8f-e456452b69ac-tls-certs" pod "6de44df9-846c-410e-aa8f-e456452b69ac" (UID: "6de44df9-846c-410e-aa8f-e456452b69ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:38.401186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.401150    5503 generic.go:356] "Generic (PLEG): container finished" podID="6de44df9-846c-410e-aa8f-e456452b69ac" containerID="c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016" exitCode=0
Oct 30 22:06:38.401361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.401194    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"
Oct 30 22:06:38.401361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.401231    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6" event={"ID":"6de44df9-846c-410e-aa8f-e456452b69ac","Type":"ContainerDied","Data":"c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016"}
Oct 30 22:06:38.401361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.401283    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6" event={"ID":"6de44df9-846c-410e-aa8f-e456452b69ac","Type":"ContainerDied","Data":"9a132abae9aaa100362b4f57b32e18ec20e36479ea9f0bf9bd7bf05716887511"}
Oct 30 22:06:38.401361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.401300    5503 scope.go:122] "RemoveContainer" containerID="c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016"
Oct 30 22:06:38.411722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.411698    5503 scope.go:122] "RemoveContainer" containerID="c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016"
Oct 30 22:06:38.412029 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:06:38.412010    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016\": container with ID starting with c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016 not found: ID does not exist" containerID="c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016"
Oct 30 22:06:38.412102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.412040    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016"} err="failed to get container status \"c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016\": rpc error: code = NotFound desc = could not find container \"c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016\": container with ID starting with c30b9dcceed8d93057c3f70ef5eed4458d5db860ecf2070950bcbc741bf50016 not found: ID does not exist"
Oct 30 22:06:38.412229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.412210    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de44df9-846c-410e-aa8f-e456452b69ac-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:38.412285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.412232    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de44df9-846c-410e-aa8f-e456452b69ac-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:38.412285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.412243    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswhn\" (UniqueName: \"kubernetes.io/projected/6de44df9-846c-410e-aa8f-e456452b69ac-kube-api-access-dswhn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:38.572693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.572609    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"]
Oct 30 22:06:38.576770 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:38.576736    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/e2e-conflict-b-d3c7fa24-kserve-79c6cfdb49-6m9v6"]
Oct 30 22:06:39.006236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.006203    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:39.739073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.739036    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"]
Oct 30 22:06:39.739536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.739522    5503 cpu_manager.go:414] "containerMap: removing container" podUID="6de44df9-846c-410e-aa8f-e456452b69ac" containerName="main"
Oct 30 22:06:39.739581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.739538    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de44df9-846c-410e-aa8f-e456452b69ac" containerName="main"
Oct 30 22:06:39.739620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.739607    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de44df9-846c-410e-aa8f-e456452b69ac" containerName="main"
Oct 30 22:06:39.742298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.742277    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.795582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.795541    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de44df9-846c-410e-aa8f-e456452b69ac" path="/var/lib/kubelet/pods/6de44df9-846c-410e-aa8f-e456452b69ac/volumes"
Oct 30 22:06:39.799128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.799098    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"]
Oct 30 22:06:39.839327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.839286    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.839327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.839331    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssdq\" (UniqueName: \"kubernetes.io/projected/83c1f5e8-504a-4764-a7db-fdfe9444c58f-kube-api-access-nssdq\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.839532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.839416    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.845871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.845851    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:39.845923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.845900    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:39.846306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.846295    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:39.846455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.846444    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:06:39.940620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.940583    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.940810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.940765    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.940810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.940798    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssdq\" (UniqueName: \"kubernetes.io/projected/83c1f5e8-504a-4764-a7db-fdfe9444c58f-kube-api-access-nssdq\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.941470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.941442    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:39.942954 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:39.942935    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:40.003448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:40.003381    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssdq\" (UniqueName: \"kubernetes.io/projected/83c1f5e8-504a-4764-a7db-fdfe9444c58f-kube-api-access-nssdq\") pod \"maas-api-revoke-sub-9d7d2de1-4dj79\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:40.055113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:40.055084    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:40.427682 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:06:40.427629    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c1f5e8_504a_4764_a7db_fdfe9444c58f.slice/crio-1f7be7cbda52b3ff1643005a2adeb0adb2e72c61cbc55edf5888f33d75cbf96a WatchSource:0}: Error finding container 1f7be7cbda52b3ff1643005a2adeb0adb2e72c61cbc55edf5888f33d75cbf96a: Status 404 returned error can't find the container with id 1f7be7cbda52b3ff1643005a2adeb0adb2e72c61cbc55edf5888f33d75cbf96a
Oct 30 22:06:40.430663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:40.430614    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"]
Oct 30 22:06:41.230304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:41.230167    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:41.419527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:41.419491    5503 generic.go:356] "Generic (PLEG): container finished" podID="83c1f5e8-504a-4764-a7db-fdfe9444c58f" containerID="85377f54d361500924c818fa3313e06db60cd0951fe7aea54518743c946e5376" exitCode=0
Oct 30 22:06:41.419706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:41.419575    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79" event={"ID":"83c1f5e8-504a-4764-a7db-fdfe9444c58f","Type":"ContainerDied","Data":"85377f54d361500924c818fa3313e06db60cd0951fe7aea54518743c946e5376"}
Oct 30 22:06:41.419706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:41.419626    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79" event={"ID":"83c1f5e8-504a-4764-a7db-fdfe9444c58f","Type":"ContainerStarted","Data":"1f7be7cbda52b3ff1643005a2adeb0adb2e72c61cbc55edf5888f33d75cbf96a"}
Oct 30 22:06:41.444410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:41.444365    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:41.647165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:41.647071    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:42.666832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.666806    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:42.779158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.779069    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-openshift-ca\") pod \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") "
Oct 30 22:06:42.779158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.779113    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c1f5e8-504a-4764-a7db-fdfe9444c58f-kube-api-access-nssdq\" (UniqueName: \"kubernetes.io/projected/83c1f5e8-504a-4764-a7db-fdfe9444c58f-kube-api-access-nssdq\") pod \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") "
Oct 30 22:06:42.779403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.779200    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-xks-ca\") pod \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\" (UID: \"83c1f5e8-504a-4764-a7db-fdfe9444c58f\") "
Oct 30 22:06:42.779501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.779469    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-openshift-ca" pod "83c1f5e8-504a-4764-a7db-fdfe9444c58f" (UID: "83c1f5e8-504a-4764-a7db-fdfe9444c58f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:42.779604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.779590    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:42.781514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.781482    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c1f5e8-504a-4764-a7db-fdfe9444c58f-kube-api-access-nssdq" pod "83c1f5e8-504a-4764-a7db-fdfe9444c58f" (UID: "83c1f5e8-504a-4764-a7db-fdfe9444c58f"). InnerVolumeSpecName "kube-api-access-nssdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:42.781659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.781540    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-xks-ca" pod "83c1f5e8-504a-4764-a7db-fdfe9444c58f" (UID: "83c1f5e8-504a-4764-a7db-fdfe9444c58f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:42.880537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.880461    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c1f5e8-504a-4764-a7db-fdfe9444c58f-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:42.880537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:42.880494    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nssdq\" (UniqueName: \"kubernetes.io/projected/83c1f5e8-504a-4764-a7db-fdfe9444c58f-kube-api-access-nssdq\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:43.432272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:43.432234    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79" event={"ID":"83c1f5e8-504a-4764-a7db-fdfe9444c58f","Type":"ContainerDied","Data":"1f7be7cbda52b3ff1643005a2adeb0adb2e72c61cbc55edf5888f33d75cbf96a"}
Oct 30 22:06:43.432272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:43.432253    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"
Oct 30 22:06:43.432482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:43.432289    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f7be7cbda52b3ff1643005a2adeb0adb2e72c61cbc55edf5888f33d75cbf96a"
Oct 30 22:06:43.558704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:43.558672    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:48.137390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:48.137324    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:50.238466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:50.237458    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:06:54.020498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:54.020441    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:56.800118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.800084    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"]
Oct 30 22:06:56.800563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.800525    5503 cpu_manager.go:414] "containerMap: removing container" podUID="83c1f5e8-504a-4764-a7db-fdfe9444c58f" containerName="revoke-keys"
Oct 30 22:06:56.800563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.800537    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c1f5e8-504a-4764-a7db-fdfe9444c58f" containerName="revoke-keys"
Oct 30 22:06:56.800671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.800619    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c1f5e8-504a-4764-a7db-fdfe9444c58f" containerName="revoke-keys"
Oct 30 22:06:56.803559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.803540    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:56.859607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.859576    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"]
Oct 30 22:06:56.908060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.908027    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:56.908060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.908042    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:06:56.908060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.908061    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:06:56.908976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.908960    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:06:56.951405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.951371    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:56.951556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.951423    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:56.951556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:56.951490    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2hb\" (UniqueName: \"kubernetes.io/projected/275c7d4a-5564-433c-afb1-07f38c142238-kube-api-access-fr2hb\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.053258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.053169    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.053258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.053221    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.053471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.053263    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2hb\" (UniqueName: \"kubernetes.io/projected/275c7d4a-5564-433c-afb1-07f38c142238-kube-api-access-fr2hb\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.053931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.053912    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.055446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.055421    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.115959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.115923    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2hb\" (UniqueName: \"kubernetes.io/projected/275c7d4a-5564-433c-afb1-07f38c142238-kube-api-access-fr2hb\") pod \"maas-api-revoke-sub-8256e71c-ldkd8\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.116114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.116082    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:57.486863 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:06:57.486834    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275c7d4a_5564_433c_afb1_07f38c142238.slice/crio-9dd4926323bd426e890670df5de53d31806cf6f22a8a3549fc8a74a890b84ebc WatchSource:0}: Error finding container 9dd4926323bd426e890670df5de53d31806cf6f22a8a3549fc8a74a890b84ebc: Status 404 returned error can't find the container with id 9dd4926323bd426e890670df5de53d31806cf6f22a8a3549fc8a74a890b84ebc
Oct 30 22:06:57.489396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.489365    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"]
Oct 30 22:06:57.544577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:57.544546    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8" event={"ID":"275c7d4a-5564-433c-afb1-07f38c142238","Type":"ContainerStarted","Data":"9dd4926323bd426e890670df5de53d31806cf6f22a8a3549fc8a74a890b84ebc"}
Oct 30 22:06:58.098573 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:06:58.098489    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:06:58.551954 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:58.551925    5503 generic.go:356] "Generic (PLEG): container finished" podID="275c7d4a-5564-433c-afb1-07f38c142238" containerID="b925c310d8025298bbce31b97b05a51d4e0781be40f22f54d5d4142cdd53605b" exitCode=0
Oct 30 22:06:58.552149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:58.552011    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8" event={"ID":"275c7d4a-5564-433c-afb1-07f38c142238","Type":"ContainerDied","Data":"b925c310d8025298bbce31b97b05a51d4e0781be40f22f54d5d4142cdd53605b"}
Oct 30 22:06:58.741449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:58.741415    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:59.703926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.703895    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:06:59.782687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.782619    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-xks-ca\") pod \"275c7d4a-5564-433c-afb1-07f38c142238\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") "
Oct 30 22:06:59.782899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.782736    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-openshift-ca\") pod \"275c7d4a-5564-433c-afb1-07f38c142238\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") "
Oct 30 22:06:59.782899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.782779    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275c7d4a-5564-433c-afb1-07f38c142238-kube-api-access-fr2hb\" (UniqueName: \"kubernetes.io/projected/275c7d4a-5564-433c-afb1-07f38c142238-kube-api-access-fr2hb\") pod \"275c7d4a-5564-433c-afb1-07f38c142238\" (UID: \"275c7d4a-5564-433c-afb1-07f38c142238\") "
Oct 30 22:06:59.783191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.783156    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-openshift-ca" pod "275c7d4a-5564-433c-afb1-07f38c142238" (UID: "275c7d4a-5564-433c-afb1-07f38c142238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:59.784769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.784743    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-xks-ca" pod "275c7d4a-5564-433c-afb1-07f38c142238" (UID: "275c7d4a-5564-433c-afb1-07f38c142238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:59.784895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.784839    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275c7d4a-5564-433c-afb1-07f38c142238-kube-api-access-fr2hb" pod "275c7d4a-5564-433c-afb1-07f38c142238" (UID: "275c7d4a-5564-433c-afb1-07f38c142238"). InnerVolumeSpecName "kube-api-access-fr2hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:59.883946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.883863    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:59.883946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.883900    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275c7d4a-5564-433c-afb1-07f38c142238-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:59.883946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:06:59.883915    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2hb\" (UniqueName: \"kubernetes.io/projected/275c7d4a-5564-433c-afb1-07f38c142238-kube-api-access-fr2hb\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:00.564419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:00.564386    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"
Oct 30 22:07:00.564575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:00.564430    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8" event={"ID":"275c7d4a-5564-433c-afb1-07f38c142238","Type":"ContainerDied","Data":"9dd4926323bd426e890670df5de53d31806cf6f22a8a3549fc8a74a890b84ebc"}
Oct 30 22:07:00.564575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:00.564475    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dd4926323bd426e890670df5de53d31806cf6f22a8a3549fc8a74a890b84ebc"
Oct 30 22:07:03.329021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:03.328983    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:03.931174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:03.931145    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:07:03.937451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:03.937429    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:07:07.947171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:07.947132    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:08.769135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:08.769100    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:07:09.096861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.096771    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"]
Oct 30 22:07:09.097876 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.097856    5503 cpu_manager.go:414] "containerMap: removing container" podUID="275c7d4a-5564-433c-afb1-07f38c142238" containerName="revoke-keys"
Oct 30 22:07:09.098005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.097992    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275c7d4a-5564-433c-afb1-07f38c142238" containerName="revoke-keys"
Oct 30 22:07:09.098278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.098263    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="275c7d4a-5564-433c-afb1-07f38c142238" containerName="revoke-keys"
Oct 30 22:07:09.106254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.106231    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.194984    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195034    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195131    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195187    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195243    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195311    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195433 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195378    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195532    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkx54\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-kube-api-access-wkx54\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.195594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.195580    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.213705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.213673    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"]
Oct 30 22:07:09.234629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.234595    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e65019-openshift-default-dockercfg-jjq2k\""
Oct 30 22:07:09.297501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.297456    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.297953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.297926    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.298170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.298153    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkx54\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-kube-api-access-wkx54\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.298298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.298279    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.298491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.298473    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.298702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.298683    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.298897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.298880    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.299025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.299005    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.299135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.299121    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.299507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.299481    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.299669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.299597    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.300031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.300003    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.300490 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.300469    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.300992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.300686    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.313366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.313341    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.316000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.315940    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.363341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.363256    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkx54\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-kube-api-access-wkx54\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.368342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.368314    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.422403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.422364    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:09.896864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.896822    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"]
Oct 30 22:07:09.934733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.934683    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:09.934862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.934778    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:09.934862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:09.934827    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:10.407715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.407662    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"]
Oct 30 22:07:10.428232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.428204    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.535283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.535238    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"]
Oct 30 22:07:10.546393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.546349    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.566209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.566177    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xw8\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-kube-api-access-g4xw8\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.566524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.566500    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.566752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.566733    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.566945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.566927    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.823121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.823041    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.823492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.823466    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.823725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.823705    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.823975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.823957    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.824191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.821764    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" event={"ID":"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5","Type":"ContainerStarted","Data":"97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e"}
Oct 30 22:07:10.824358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.824337    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" event={"ID":"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5","Type":"ContainerStarted","Data":"da30d9254eea35723a2c89bc622eac7a71420802cc716861ea2fa95b11b5fb89"}
Oct 30 22:07:10.927331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.927289    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.927677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.927651    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xw8\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-kube-api-access-g4xw8\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.927838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.927821    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.928003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.927983    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.928130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.928114    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.928306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.928290    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.928469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.928451    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.928654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.928621    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.943844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.943810    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.944544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.944519    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-data\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.946889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.944954    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.947022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.945180    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-credential-socket\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.947407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.947382    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-certs\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.948806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.948757    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podStartSLOduration=1.948741469 podStartE2EDuration="1.948741469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:10.948067097 +0000 UTC m=+1207.282643867" watchObservedRunningTime="2026-10-30 22:07:10.948741469 +0000 UTC m=+1207.283318238"
Oct 30 22:07:10.954652 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.950616    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istiod-ca-cert\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.989659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.972105    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-envoy\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:10.989659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:10.983547    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-podinfo\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:11.028292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.028223    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-token\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:11.029110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.029085    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xw8\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-kube-api-access-g4xw8\") pod \"e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") " pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:11.063879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.063843    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:11.235785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.235735    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:11.277210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.276838    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:11.426706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.424740    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:11.427307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.426674    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:11.427499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.427464    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:11.670661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.662002    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:11.721597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.721252    5503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:11.825687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.797566    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:11.871680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.869756    5503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:11.916009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:11.914780    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:12.152931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.152898    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"]
Oct 30 22:07:12.245672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.237696    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:12.245672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.237803    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:12.245672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.237854    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:12.346907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.346783    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:12.348093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.348016    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:07:12.383010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.382839    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:12.424660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.423073    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:12.424660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.423151    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:12.501694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.500995    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:07:12.502653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.502243    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:12.632319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.630473    5503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:12.686727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.676760    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:12.688180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.687030    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:07:12.802774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.802729    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:12.803410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:12.803384    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:07:13.198673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.194850    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" event={"ID":"138d9a0c-5eb8-4daf-9b29-ed780c71d938","Type":"ContainerStarted","Data":"cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118"}
Oct 30 22:07:13.237273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.237226    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" event={"ID":"138d9a0c-5eb8-4daf-9b29-ed780c71d938","Type":"ContainerStarted","Data":"ce59566e5182fd50f938e248cb3d89067e45ff9b0b7ad844666f2dfe107e460c"}
Oct 30 22:07:13.245759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.237132    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:13.283660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.270912    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:07:13.388780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.388733    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:13.389794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.389767    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:07:13.391215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.389992    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:13.413112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.413085    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:07:13.430660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.426027    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:13.430660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.426078    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:13.515709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.510963    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:13.515709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.512347    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:07:13.515709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.513495    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podStartSLOduration=3.513478025 podStartE2EDuration="3.513478025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:13.363181307 +0000 UTC m=+1209.697758074" watchObservedRunningTime="2026-10-30 22:07:13.513478025 +0000 UTC m=+1209.848054795"
Oct 30 22:07:13.733992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.733946    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:07:13.925161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:13.925129    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:07:14.075074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:14.075025    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:14.089926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:14.089893    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:07:14.093829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:14.093761    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 22:07:14.106349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:14.106152    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:07:14.441660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:14.439036    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:14.441660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:14.439105    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:14.739155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:14.739067    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:15.068543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:15.068016    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:07:15.068543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:15.068078    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 22:07:15.432171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:15.432134    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:15.432455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:15.432426    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:15.603055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:15.603022    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:16.066665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:16.066045    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:07:16.066665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:16.066109    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 22:07:16.088662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:16.086729    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:16.428668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:16.428468    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:16.428668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:16.428526    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:17.065056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:17.064990    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:07:17.065272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:17.065090    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 22:07:17.432661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:17.428994    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:17.432661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:17.429057    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:18.065057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:18.065016    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:07:18.065288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:18.065084    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 22:07:18.422875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:18.422841    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 30 22:07:18.423159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:18.423129    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 30 22:07:19.065603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:19.065567    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Oct 30 22:07:19.066025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:19.065630    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Oct 30 22:07:19.426938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:19.426908    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:19.427352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:19.427324    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:19.428015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:19.427994    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:07:19.457849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:19.457818    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:20.288864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:20.288808    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:20.289350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:20.289263    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:20.290407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:20.290382    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:07:21.385808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.385775    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"]
Oct 30 22:07:21.388579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.388549    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.445314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.445279    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"]
Oct 30 22:07:21.482722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.482679    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhsn\" (UniqueName: \"kubernetes.io/projected/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-kube-api-access-qxhsn\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.482884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.482779    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.482884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.482865    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.493040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.493013    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:21.493157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.493037    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:21.493157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.493147    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:21.493224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.493213    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:07:21.583718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.583681    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhsn\" (UniqueName: \"kubernetes.io/projected/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-kube-api-access-qxhsn\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.583910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.583749    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.583910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.583779    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.584436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.584412    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.586222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.586200    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.646372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.646293    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhsn\" (UniqueName: \"kubernetes.io/projected/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-kube-api-access-qxhsn\") pod \"maas-api-revoke-sub-07dd7c9a-fswfc\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:21.704496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:21.704459    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:22.074956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:22.074930    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"]
Oct 30 22:07:22.229536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:22.229492    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc" event={"ID":"38cc1457-a4fe-40fe-8ff6-40321e78ed5e","Type":"ContainerStarted","Data":"12289cc6f8a890545cc89e489bc60cd804a417cf3e68fc5fea3ebfc23e61e213"}
Oct 30 22:07:23.237427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:23.237392    5503 generic.go:356] "Generic (PLEG): container finished" podID="38cc1457-a4fe-40fe-8ff6-40321e78ed5e" containerID="06ce2d17b83a9a90898c730f78be4edac1342cb1c8c598b8dec5e7c9acfb3d6b" exitCode=0
Oct 30 22:07:23.237826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:23.237433    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc" event={"ID":"38cc1457-a4fe-40fe-8ff6-40321e78ed5e","Type":"ContainerDied","Data":"06ce2d17b83a9a90898c730f78be4edac1342cb1c8c598b8dec5e7c9acfb3d6b"}
Oct 30 22:07:24.013947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.013917    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:24.381842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.381817    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:24.520845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.520814    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-xks-ca\") pod \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") "
Oct 30 22:07:24.520994 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.520862    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-kube-api-access-qxhsn\" (UniqueName: \"kubernetes.io/projected/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-kube-api-access-qxhsn\") pod \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") "
Oct 30 22:07:24.520994 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.520961    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-openshift-ca\") pod \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\" (UID: \"38cc1457-a4fe-40fe-8ff6-40321e78ed5e\") "
Oct 30 22:07:24.521411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.521378    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-openshift-ca" pod "38cc1457-a4fe-40fe-8ff6-40321e78ed5e" (UID: "38cc1457-a4fe-40fe-8ff6-40321e78ed5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:24.521532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.521495    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:24.523178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.523144    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-kube-api-access-qxhsn" pod "38cc1457-a4fe-40fe-8ff6-40321e78ed5e" (UID: "38cc1457-a4fe-40fe-8ff6-40321e78ed5e"). InnerVolumeSpecName "kube-api-access-qxhsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:24.523753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.523718    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-xks-ca" pod "38cc1457-a4fe-40fe-8ff6-40321e78ed5e" (UID: "38cc1457-a4fe-40fe-8ff6-40321e78ed5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:24.622246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.622170    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:24.622246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:24.622198    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhsn\" (UniqueName: \"kubernetes.io/projected/38cc1457-a4fe-40fe-8ff6-40321e78ed5e-kube-api-access-qxhsn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:25.249231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:25.249206    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"
Oct 30 22:07:25.249409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:25.249211    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc" event={"ID":"38cc1457-a4fe-40fe-8ff6-40321e78ed5e","Type":"ContainerDied","Data":"12289cc6f8a890545cc89e489bc60cd804a417cf3e68fc5fea3ebfc23e61e213"}
Oct 30 22:07:25.249409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:25.249343    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12289cc6f8a890545cc89e489bc60cd804a417cf3e68fc5fea3ebfc23e61e213"
Oct 30 22:07:28.102660 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:07:28.101762    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:07:29.020478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:29.020444    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:30.928119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:30.928081    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:32.120922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.120887    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"]
Oct 30 22:07:32.121397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.121380    5503 cpu_manager.go:414] "containerMap: removing container" podUID="38cc1457-a4fe-40fe-8ff6-40321e78ed5e" containerName="revoke-keys"
Oct 30 22:07:32.121444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.121399    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38cc1457-a4fe-40fe-8ff6-40321e78ed5e" containerName="revoke-keys"
Oct 30 22:07:32.121517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.121508    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="38cc1457-a4fe-40fe-8ff6-40321e78ed5e" containerName="revoke-keys"
Oct 30 22:07:32.123729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.123714    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.168554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.168522    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.168722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.168669    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.168817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.168802    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldt9c\" (UniqueName: \"kubernetes.io/projected/fcd8ad12-362c-49f6-8d64-6fda51e12f86-kube-api-access-ldt9c\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.178761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.178732    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"]
Oct 30 22:07:32.226892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.226856    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:32.226892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.226890    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:32.227091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.226898    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:07:32.227537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.227521    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:32.269463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.269429    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldt9c\" (UniqueName: \"kubernetes.io/projected/fcd8ad12-362c-49f6-8d64-6fda51e12f86-kube-api-access-ldt9c\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.269615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.269482    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.269615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.269548    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.270223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.270198    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.271908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.271884    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.331996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.331961    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldt9c\" (UniqueName: \"kubernetes.io/projected/fcd8ad12-362c-49f6-8d64-6fda51e12f86-kube-api-access-ldt9c\") pod \"maas-api-revoke-sub-7a493d19-pvp8m\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.443376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.443342    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:32.858746 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:07:32.858700    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd8ad12_362c_49f6_8d64_6fda51e12f86.slice/crio-51a0bddabcd31b0ca8a03defb768d94f16d1f2c8280103811fce28cccae981a7 WatchSource:0}: Error finding container 51a0bddabcd31b0ca8a03defb768d94f16d1f2c8280103811fce28cccae981a7: Status 404 returned error can't find the container with id 51a0bddabcd31b0ca8a03defb768d94f16d1f2c8280103811fce28cccae981a7
Oct 30 22:07:32.861177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:32.861149    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"]
Oct 30 22:07:33.358144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:33.358105    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m" event={"ID":"fcd8ad12-362c-49f6-8d64-6fda51e12f86","Type":"ContainerStarted","Data":"2ee73be4f0c23101914ef2d466ea6a44d9f3dadb6fc129263e2c8c98888d42a8"}
Oct 30 22:07:33.358528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:33.358169    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m" event={"ID":"fcd8ad12-362c-49f6-8d64-6fda51e12f86","Type":"ContainerStarted","Data":"51a0bddabcd31b0ca8a03defb768d94f16d1f2c8280103811fce28cccae981a7"}
Oct 30 22:07:33.477447 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:33.477396    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m" podStartSLOduration=1.477382565 podStartE2EDuration="1.477382565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:33.475122727 +0000 UTC m=+1229.809699495" watchObservedRunningTime="2026-10-30 22:07:33.477382565 +0000 UTC m=+1229.811959334"
Oct 30 22:07:33.619851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:33.619801    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:34.364508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:34.364468    5503 generic.go:356] "Generic (PLEG): container finished" podID="fcd8ad12-362c-49f6-8d64-6fda51e12f86" containerID="2ee73be4f0c23101914ef2d466ea6a44d9f3dadb6fc129263e2c8c98888d42a8" exitCode=0
Oct 30 22:07:34.364958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:34.364547    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m" event={"ID":"fcd8ad12-362c-49f6-8d64-6fda51e12f86","Type":"ContainerDied","Data":"2ee73be4f0c23101914ef2d466ea6a44d9f3dadb6fc129263e2c8c98888d42a8"}
Oct 30 22:07:35.501017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.500992    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:35.608856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.608820    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-openshift-ca\") pod \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") "
Oct 30 22:07:35.609037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.608895    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd8ad12-362c-49f6-8d64-6fda51e12f86-kube-api-access-ldt9c\" (UniqueName: \"kubernetes.io/projected/fcd8ad12-362c-49f6-8d64-6fda51e12f86-kube-api-access-ldt9c\") pod \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") "
Oct 30 22:07:35.609037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.608954    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-xks-ca\") pod \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\" (UID: \"fcd8ad12-362c-49f6-8d64-6fda51e12f86\") "
Oct 30 22:07:35.609217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.609192    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-openshift-ca" pod "fcd8ad12-362c-49f6-8d64-6fda51e12f86" (UID: "fcd8ad12-362c-49f6-8d64-6fda51e12f86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:35.609424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.609399    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:35.611056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.611034    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd8ad12-362c-49f6-8d64-6fda51e12f86-kube-api-access-ldt9c" pod "fcd8ad12-362c-49f6-8d64-6fda51e12f86" (UID: "fcd8ad12-362c-49f6-8d64-6fda51e12f86"). InnerVolumeSpecName "kube-api-access-ldt9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:35.611056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.611044    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-xks-ca" pod "fcd8ad12-362c-49f6-8d64-6fda51e12f86" (UID: "fcd8ad12-362c-49f6-8d64-6fda51e12f86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:35.710425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.710393    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldt9c\" (UniqueName: \"kubernetes.io/projected/fcd8ad12-362c-49f6-8d64-6fda51e12f86-kube-api-access-ldt9c\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:35.710425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:35.710428    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd8ad12-362c-49f6-8d64-6fda51e12f86-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:36.376910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:36.376879    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"
Oct 30 22:07:36.377083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:36.376879    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m" event={"ID":"fcd8ad12-362c-49f6-8d64-6fda51e12f86","Type":"ContainerDied","Data":"51a0bddabcd31b0ca8a03defb768d94f16d1f2c8280103811fce28cccae981a7"}
Oct 30 22:07:36.377083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:36.377007    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a0bddabcd31b0ca8a03defb768d94f16d1f2c8280103811fce28cccae981a7"
Oct 30 22:07:38.136600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:38.136557    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:42.748492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:42.748460    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:44.587875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:44.587838    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:07:47.876157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.876119    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"]
Oct 30 22:07:47.879664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.877786    5503 cpu_manager.go:414] "containerMap: removing container" podUID="fcd8ad12-362c-49f6-8d64-6fda51e12f86" containerName="revoke-keys"
Oct 30 22:07:47.879664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.877814    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd8ad12-362c-49f6-8d64-6fda51e12f86" containerName="revoke-keys"
Oct 30 22:07:47.879664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.877925    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd8ad12-362c-49f6-8d64-6fda51e12f86" containerName="revoke-keys"
Oct 30 22:07:47.883195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.883171    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.934892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.934851    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.935260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.935234    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.935478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.935457    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.935708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.935687    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.935943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.935925    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.936143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.936126    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.936312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.936295    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.937716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.937692    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:47.937936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.937567    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d09fff-openshift-default-dockercfg-596tc\""
Oct 30 22:07:47.938083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:47.938064    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76ts\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-kube-api-access-k76ts\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.039576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039535    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.039778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039615    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.039778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039669    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.039778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039736    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.039778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039763    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76ts\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-kube-api-access-k76ts\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039815    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039863    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039901    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.039926    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.040101    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-credential-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.040346    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-data\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.040412    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-certs\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.040614    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-socket\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.040742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.040706    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istiod-ca-cert\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.042074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.042050    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-envoy\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.042548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.042528    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-podinfo\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.085965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.085934    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"]
Oct 30 22:07:48.104947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.104913    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-token\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.105378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.105332    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76ts\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-kube-api-access-k76ts\") pod \"e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") " pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.207595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.207563    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:48.631932 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:07:48.631890    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e6b256_5f2e_4dc1_96ca_001bb509a8ad.slice/crio-a3afafe7e2dfb8795c2fbec34e30427833384da6cd0ef22667df07248278c441 WatchSource:0}: Error finding container a3afafe7e2dfb8795c2fbec34e30427833384da6cd0ef22667df07248278c441: Status 404 returned error can't find the container with id a3afafe7e2dfb8795c2fbec34e30427833384da6cd0ef22667df07248278c441
Oct 30 22:07:48.634660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.634620    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"]
Oct 30 22:07:48.634826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.634793    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:48.634897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.634872    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:48.634956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.634921    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:48.841476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:48.841443    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:49.485056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:49.485022    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" event={"ID":"99e6b256-5f2e-4dc1-96ca-001bb509a8ad","Type":"ContainerStarted","Data":"348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7"}
Oct 30 22:07:49.485490 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:49.485104    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" event={"ID":"99e6b256-5f2e-4dc1-96ca-001bb509a8ad","Type":"ContainerStarted","Data":"a3afafe7e2dfb8795c2fbec34e30427833384da6cd0ef22667df07248278c441"}
Oct 30 22:07:49.714192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:49.714140    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podStartSLOduration=2.714125711 podStartE2EDuration="2.714125711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:49.662343227 +0000 UTC m=+1245.996919997" watchObservedRunningTime="2026-10-30 22:07:49.714125711 +0000 UTC m=+1246.048702480"
Oct 30 22:07:50.208043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.208009    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:50.209698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.209631    5503 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:07:50.209829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.209736    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:07:50.343279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.343243    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"]
Oct 30 22:07:50.346974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.346957    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.369068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.369038    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.369192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.369093    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.369235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.369199    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzv56\" (UniqueName: \"kubernetes.io/projected/047e60c4-077a-4450-b26b-f239cbf5d900-kube-api-access-jzv56\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.400522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.400487    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:50.400522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.400501    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:07:50.452097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.452061    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:50.452293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.452109    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:50.454870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.454845    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"]
Oct 30 22:07:50.470094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.470025    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.470094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.470090    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.470290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.470141    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzv56\" (UniqueName: \"kubernetes.io/projected/047e60c4-077a-4450-b26b-f239cbf5d900-kube-api-access-jzv56\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.470745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.470720    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.472596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.472572    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.542471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.542430    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzv56\" (UniqueName: \"kubernetes.io/projected/047e60c4-077a-4450-b26b-f239cbf5d900-kube-api-access-jzv56\") pod \"maas-api-revoke-sub-2ba405f9-4b9hn\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:50.660603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:50.660571    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:51.045991 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:07:51.045958    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047e60c4_077a_4450_b26b_f239cbf5d900.slice/crio-e2ad973080af433ccb9eb63743faaf427c4b86f389eeff7f6fa35bcd081115c2 WatchSource:0}: Error finding container e2ad973080af433ccb9eb63743faaf427c4b86f389eeff7f6fa35bcd081115c2: Status 404 returned error can't find the container with id e2ad973080af433ccb9eb63743faaf427c4b86f389eeff7f6fa35bcd081115c2
Oct 30 22:07:51.048064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:51.048035    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"]
Oct 30 22:07:51.208934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:51.208896    5503 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:07:51.209126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:51.208960    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:07:51.498579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:51.498533    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn" event={"ID":"047e60c4-077a-4450-b26b-f239cbf5d900","Type":"ContainerStarted","Data":"ec7123824665d03596bd4226aa7dcbc4b7ac1a892f84e68ead1d44e17da1bddc"}
Oct 30 22:07:51.498777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:51.498593    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn" event={"ID":"047e60c4-077a-4450-b26b-f239cbf5d900","Type":"ContainerStarted","Data":"e2ad973080af433ccb9eb63743faaf427c4b86f389eeff7f6fa35bcd081115c2"}
Oct 30 22:07:51.640047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:51.639991    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn" podStartSLOduration=1.6399753339999998 podStartE2EDuration="1.639975334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:51.638113816 +0000 UTC m=+1247.972690586" watchObservedRunningTime="2026-10-30 22:07:51.639975334 +0000 UTC m=+1247.974552103"
Oct 30 22:07:52.208952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:52.208917    5503 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:07:52.209100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:52.209004    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:07:52.506026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:52.505931    5503 generic.go:356] "Generic (PLEG): container finished" podID="047e60c4-077a-4450-b26b-f239cbf5d900" containerID="ec7123824665d03596bd4226aa7dcbc4b7ac1a892f84e68ead1d44e17da1bddc" exitCode=0
Oct 30 22:07:52.506188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:52.506013    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn" event={"ID":"047e60c4-077a-4450-b26b-f239cbf5d900","Type":"ContainerDied","Data":"ec7123824665d03596bd4226aa7dcbc4b7ac1a892f84e68ead1d44e17da1bddc"}
Oct 30 22:07:53.208524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.208488    5503 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:07:53.208975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.208554    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:07:53.572140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.572105    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:53.807895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.807867    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:53.947185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.947147    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047e60c4-077a-4450-b26b-f239cbf5d900-kube-api-access-jzv56\" (UniqueName: \"kubernetes.io/projected/047e60c4-077a-4450-b26b-f239cbf5d900-kube-api-access-jzv56\") pod \"047e60c4-077a-4450-b26b-f239cbf5d900\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") "
Oct 30 22:07:53.947408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.947264    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-openshift-ca\") pod \"047e60c4-077a-4450-b26b-f239cbf5d900\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") "
Oct 30 22:07:53.947485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.947470    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-xks-ca\") pod \"047e60c4-077a-4450-b26b-f239cbf5d900\" (UID: \"047e60c4-077a-4450-b26b-f239cbf5d900\") "
Oct 30 22:07:53.947732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.947700    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-openshift-ca" pod "047e60c4-077a-4450-b26b-f239cbf5d900" (UID: "047e60c4-077a-4450-b26b-f239cbf5d900"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:53.947929 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.947911    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:53.949560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.949532    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047e60c4-077a-4450-b26b-f239cbf5d900-kube-api-access-jzv56" pod "047e60c4-077a-4450-b26b-f239cbf5d900" (UID: "047e60c4-077a-4450-b26b-f239cbf5d900"). InnerVolumeSpecName "kube-api-access-jzv56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:53.949693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:53.949591    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-xks-ca" pod "047e60c4-077a-4450-b26b-f239cbf5d900" (UID: "047e60c4-077a-4450-b26b-f239cbf5d900"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:54.048968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.048868    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/047e60c4-077a-4450-b26b-f239cbf5d900-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:54.048968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.048907    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzv56\" (UniqueName: \"kubernetes.io/projected/047e60c4-077a-4450-b26b-f239cbf5d900-kube-api-access-jzv56\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:54.208556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.208522    5503 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:07:54.208962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.208581    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:07:54.447656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.447571    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"]
Oct 30 22:07:54.451663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.448497    5503 cpu_manager.go:414] "containerMap: removing container" podUID="047e60c4-077a-4450-b26b-f239cbf5d900" containerName="revoke-keys"
Oct 30 22:07:54.451663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.448524    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047e60c4-077a-4450-b26b-f239cbf5d900" containerName="revoke-keys"
Oct 30 22:07:54.451663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.448711    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="047e60c4-077a-4450-b26b-f239cbf5d900" containerName="revoke-keys"
Oct 30 22:07:54.452699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.452672    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.531357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.531307    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn" event={"ID":"047e60c4-077a-4450-b26b-f239cbf5d900","Type":"ContainerDied","Data":"e2ad973080af433ccb9eb63743faaf427c4b86f389eeff7f6fa35bcd081115c2"}
Oct 30 22:07:54.531654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.531621    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ad973080af433ccb9eb63743faaf427c4b86f389eeff7f6fa35bcd081115c2"
Oct 30 22:07:54.531831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.531553    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"
Oct 30 22:07:54.554052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554017    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554065    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554151    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554180    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694c451-7d40-4d19-a63f-4d62c52c734b-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554238    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554284    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554310    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554402    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgnlx\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-kube-api-access-jgnlx\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.554547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.554465    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.575362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.575314    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"]
Oct 30 22:07:54.692020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.691987    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.692556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.692518    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.706892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.706786    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.706892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.706849    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.706892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.706874    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694c451-7d40-4d19-a63f-4d62c52c734b-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.707176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.706939    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.707176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.706978    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.707176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.707032    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.707322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.707199    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgnlx\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-kube-api-access-jgnlx\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.707322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.707286    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.708112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.708045    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.708365 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.708340    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694c451-7d40-4d19-a63f-4d62c52c734b-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.708816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.708790    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.709336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.709316    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.711988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.711955    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.748660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.747221    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.777219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.777184    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:54.778083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.778020    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.780734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.780708    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgnlx\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-kube-api-access-jgnlx\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:54.938535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:54.938502    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:55.070404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.070302    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:55.095164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.095133    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:55.208766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.208725    5503 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:07:55.209140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.208796    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:07:55.524690 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:07:55.522089    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf694c451_7d40_4d19_a63f_4d62c52c734b.slice/crio-fb967caad6bffec6d851670ad9f3c11e5248adda2b79ca208f659037b49f82fd WatchSource:0}: Error finding container fb967caad6bffec6d851670ad9f3c11e5248adda2b79ca208f659037b49f82fd: Status 404 returned error can't find the container with id fb967caad6bffec6d851670ad9f3c11e5248adda2b79ca208f659037b49f82fd
Oct 30 22:07:55.525063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.525035    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"]
Oct 30 22:07:55.525182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.525140    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:55.525252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.525234    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:55.525317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.525287    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:07:55.538905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:55.538872    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" event={"ID":"f694c451-7d40-4d19-a63f-4d62c52c734b","Type":"ContainerStarted","Data":"fb967caad6bffec6d851670ad9f3c11e5248adda2b79ca208f659037b49f82fd"}
Oct 30 22:07:56.213199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:56.213171    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:56.213722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:56.213606    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:56.214402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:56.214381    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:07:56.548286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:56.548189    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" event={"ID":"f694c451-7d40-4d19-a63f-4d62c52c734b","Type":"ContainerStarted","Data":"b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537"}
Oct 30 22:07:57.071023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:57.070987    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:07:57.072350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:57.072322    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:07:57.072494 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:57.072379    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:07:58.071082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:58.071044    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:07:58.071568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:58.071119    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:07:58.098387 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:07:58.098354    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:07:58.246795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:58.246734    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:07:59.070777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.070741    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:07:59.070970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.070800    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:07:59.442500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.442444    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podStartSLOduration=5.442429345 podStartE2EDuration="5.442429345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:56.688288476 +0000 UTC m=+1253.022865246" watchObservedRunningTime="2026-10-30 22:07:59.442429345 +0000 UTC m=+1255.777006119"
Oct 30 22:07:59.443371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.443353    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"]
Oct 30 22:07:59.447311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.447297    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.503467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.503439    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"]
Oct 30 22:07:59.544099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.544068    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.544289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.544201    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.544344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.544332    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm8vb\" (UniqueName: \"kubernetes.io/projected/7e0b508c-336f-486a-aad9-5b530c889cff-kube-api-access-hm8vb\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.549162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.549131    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:59.549162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.549149    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:07:59.549162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.549149    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:07:59.551368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.551350    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:07:59.645918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.645880    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.646073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.645983    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.646146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.646111    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm8vb\" (UniqueName: \"kubernetes.io/projected/7e0b508c-336f-486a-aad9-5b530c889cff-kube-api-access-hm8vb\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.646694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.646667    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.648173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.648147    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.709198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.709128    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm8vb\" (UniqueName: \"kubernetes.io/projected/7e0b508c-336f-486a-aad9-5b530c889cff-kube-api-access-hm8vb\") pod \"maas-api-revoke-sub-7dbd9815-8tdrt\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:07:59.760594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:07:59.760560    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:08:00.071524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:00.071442    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:00.071524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:00.071498    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:00.167729 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:08:00.167684    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf WatchSource:0}: Error finding container f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf: Status 404 returned error can't find the container with id f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf
Oct 30 22:08:00.170068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:00.170041    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"]
Oct 30 22:08:00.575229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:00.575188    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt" event={"ID":"7e0b508c-336f-486a-aad9-5b530c889cff","Type":"ContainerStarted","Data":"fee6c0bd1a321c60a402eecfaaeeff0a6146366f44c1535f587dbac03093ab3a"}
Oct 30 22:08:00.575621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:00.575245    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt" event={"ID":"7e0b508c-336f-486a-aad9-5b530c889cff","Type":"ContainerStarted","Data":"f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf"}
Oct 30 22:08:00.695546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:00.695494    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt" podStartSLOduration=1.6954772359999999 podStartE2EDuration="1.695477236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:00.694012948 +0000 UTC m=+1257.028589717" watchObservedRunningTime="2026-10-30 22:08:00.695477236 +0000 UTC m=+1257.030054005"
Oct 30 22:08:01.070929 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:01.070846    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:01.070929 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:01.070906    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:01.582741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:01.582703    5503 generic.go:356] "Generic (PLEG): container finished" podID="7e0b508c-336f-486a-aad9-5b530c889cff" containerID="fee6c0bd1a321c60a402eecfaaeeff0a6146366f44c1535f587dbac03093ab3a" exitCode=0
Oct 30 22:08:01.583142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:01.582788    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt" event={"ID":"7e0b508c-336f-486a-aad9-5b530c889cff","Type":"ContainerDied","Data":"fee6c0bd1a321c60a402eecfaaeeff0a6146366f44c1535f587dbac03093ab3a"}
Oct 30 22:08:02.071269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:02.071229    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:08:02.071456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:02.071317    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:08:02.936660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:02.930259    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:02.943288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:02.942965    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:08:03.125787 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.113913    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0b508c-336f-486a-aad9-5b530c889cff-kube-api-access-hm8vb\" (UniqueName: \"kubernetes.io/projected/7e0b508c-336f-486a-aad9-5b530c889cff-kube-api-access-hm8vb\") pod \"7e0b508c-336f-486a-aad9-5b530c889cff\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") "
Oct 30 22:08:03.125787 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.114104    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-xks-ca\") pod \"7e0b508c-336f-486a-aad9-5b530c889cff\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") "
Oct 30 22:08:03.125787 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.114170    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-openshift-ca\") pod \"7e0b508c-336f-486a-aad9-5b530c889cff\" (UID: \"7e0b508c-336f-486a-aad9-5b530c889cff\") "
Oct 30 22:08:03.127400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.127365    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-openshift-ca" pod "7e0b508c-336f-486a-aad9-5b530c889cff" (UID: "7e0b508c-336f-486a-aad9-5b530c889cff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:03.199195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.189615    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-xks-ca" pod "7e0b508c-336f-486a-aad9-5b530c889cff" (UID: "7e0b508c-336f-486a-aad9-5b530c889cff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:03.232730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.227496    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b508c-336f-486a-aad9-5b530c889cff-kube-api-access-hm8vb" pod "7e0b508c-336f-486a-aad9-5b530c889cff" (UID: "7e0b508c-336f-486a-aad9-5b530c889cff"). InnerVolumeSpecName "kube-api-access-hm8vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:03.232730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.228445    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm8vb\" (UniqueName: \"kubernetes.io/projected/7e0b508c-336f-486a-aad9-5b530c889cff-kube-api-access-hm8vb\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.232730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.228473    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.232730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.228488    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0b508c-336f-486a-aad9-5b530c889cff-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:03.623091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.623047    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt" event={"ID":"7e0b508c-336f-486a-aad9-5b530c889cff","Type":"ContainerDied","Data":"f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf"}
Oct 30 22:08:03.623603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.623583    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf"
Oct 30 22:08:03.623798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:03.623469    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"
Oct 30 22:08:04.098856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:04.098727    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:08:04.098856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:04.098803    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:04.444430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:04.443736    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:04.505616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:04.505573    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:04.874026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:04.873927    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:08:04.875290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:04.875255    5503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:04.877545 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:04.877513    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:05.012659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.010781    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:05.061660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.057959    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"]
Oct 30 22:08:05.109795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.109749    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:08:05.110416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.110389    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:05.111861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.110327    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:05.112959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.112940    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b90db6-ps6cz"]
Oct 30 22:08:05.167985 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:05.167941    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:05.254769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.254719    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:05.275582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.274714    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:08:05.448854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.448743    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:05.451301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.450580    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:08:05.498809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.498773    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:08:05.499109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.499064    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:05.520989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.519751    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:05.520989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:05.520168    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:08:06.152526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.152489    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d988eafd-fdaf-40a8-8466-97a2b28259c0" path="/var/lib/kubelet/pods/d988eafd-fdaf-40a8-8466-97a2b28259c0/volumes"
Oct 30 22:08:06.165310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.159292    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:06.175567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.164765    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:08:06.175891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.175843    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:06.188672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.180793    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:08:06.259670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.240366    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:08:06.491713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.432998    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:06.491713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.433655    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:08:06.491713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.435359    5503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:06.491713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.467756    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 30 22:08:06.706712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.601888    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:08:06.706712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.646683    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:08:06.818177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.805047    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:08:06.876890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:06.876856    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:08:07.017992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:07.017962    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:08:07.019766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:07.019745    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:08:07.029281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:07.027987    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:08:08.729324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:08.729296    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:08.892672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:08.892614    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:09.069327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:09.069208    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:09.784087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:09.784053    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:09.947713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:09.947679    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:12.436623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:12.436585    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:13.218005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:13.217966    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:13.373773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:13.373739    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:13.550686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:13.550587    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:14.296113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:14.296078    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:15.055143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:15.055107    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:15.239657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:15.239433    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:15.339428 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:15.339336    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:16.416177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:16.416140    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.474027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:17.473991    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.641488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:17.641453    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:17.789170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:17.789091    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:18.263619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:18.263588    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:18.655726 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:18.655685    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:18.656200 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:18.655698    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:19.476357 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:19.476311    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:20.176991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:20.176960    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:20.339935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:20.339904    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:20.496438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:20.496360    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:20.763260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:20.763180    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:20.969382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:20.969350    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.318463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.318427    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"]
Oct 30 22:08:23.318908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.318895    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b508c-336f-486a-aad9-5b530c889cff" containerName="revoke-keys"
Oct 30 22:08:23.318952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.318909    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b508c-336f-486a-aad9-5b530c889cff" containerName="revoke-keys"
Oct 30 22:08:23.319047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.319038    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b508c-336f-486a-aad9-5b530c889cff" containerName="revoke-keys"
Oct 30 22:08:23.322170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.322153    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.377760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.377729    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"]
Oct 30 22:08:23.401715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.401686    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7f66ab\"/\"distinct-2-w2-7f66ab-kserve-self-signed-certs\""
Oct 30 22:08:23.425314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.425279    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.441015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.440987    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-home\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.441143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.441107    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-tmp-dir\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.441197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.441156    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-dshm\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.441335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.441301    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t77z\" (UniqueName: \"kubernetes.io/projected/8bc93625-5e4f-4e54-b50c-87731dc961e6-kube-api-access-8t77z\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.441394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.441377    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc93625-5e4f-4e54-b50c-87731dc961e6-tls-certs\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.441450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.441436    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-model-cache\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.544396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.544352    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc93625-5e4f-4e54-b50c-87731dc961e6-tls-certs\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.545215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.545188    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-model-cache\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.546330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.546309    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-home\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.546930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.546910    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-tmp-dir\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.547439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.547421    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-dshm\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.548041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.548020    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t77z\" (UniqueName: \"kubernetes.io/projected/8bc93625-5e4f-4e54-b50c-87731dc961e6-kube-api-access-8t77z\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.548538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.547302    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-tmp-dir\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.548679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.546752    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-home\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.548784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.546204    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-model-cache\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.563461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.563420    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc93625-5e4f-4e54-b50c-87731dc961e6-tls-certs\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.564862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.564835    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-dshm\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.581201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.581131    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:23.612048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.612013    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t77z\" (UniqueName: \"kubernetes.io/projected/8bc93625-5e4f-4e54-b50c-87731dc961e6-kube-api-access-8t77z\") pod \"distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") " pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:23.634434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:23.634398    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:24.013410 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:08:24.013377    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bc93625_5e4f_4e54_b50c_87731dc961e6.slice/crio-288a602bca0cc062dcfe123caa35c968711126061fa04fb99e28dc8b954ffe83 WatchSource:0}: Error finding container 288a602bca0cc062dcfe123caa35c968711126061fa04fb99e28dc8b954ffe83: Status 404 returned error can't find the container with id 288a602bca0cc062dcfe123caa35c968711126061fa04fb99e28dc8b954ffe83
Oct 30 22:08:24.018961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:24.018927    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"]
Oct 30 22:08:24.028570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:24.028542    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv" event={"ID":"8bc93625-5e4f-4e54-b50c-87731dc961e6","Type":"ContainerStarted","Data":"288a602bca0cc062dcfe123caa35c968711126061fa04fb99e28dc8b954ffe83"}
Oct 30 22:08:25.036044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:25.036004    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv" event={"ID":"8bc93625-5e4f-4e54-b50c-87731dc961e6","Type":"ContainerStarted","Data":"1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f"}
Oct 30 22:08:25.163801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:25.163752    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv" podStartSLOduration=2.163738947 podStartE2EDuration="2.163738947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:25.160566395 +0000 UTC m=+1281.495143165" watchObservedRunningTime="2026-10-30 22:08:25.163738947 +0000 UTC m=+1281.498315716"
Oct 30 22:08:25.406692 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:25.406658    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:25.634733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:25.634702    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:25.905928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:25.905896    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:26.103670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:26.103616    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:27.015311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:27.015279    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:08:28.189678 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:28.189585    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:28.197873 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:28.197840    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:08:29.275042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.273796    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:29.288173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.287800    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:29.417825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.417768    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:29.603828 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.603722    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:29.651794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.650919    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:08:29.824837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.824795    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:29.866868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.866776    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:08:29.867134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.867101    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:29.868788 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:29.868760    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Oct 30 22:08:30.289753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:30.289704    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:30.290756 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:30.290732    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:08:30.316484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:30.316444    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:08:33.607729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:33.607691    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:33.635529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:33.635489    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:33.732330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:33.729791    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:34.250332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:34.250302    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:08:34.537625 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:34.537547    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:35.419153 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:35.419111    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:38.110533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:38.110493    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:42.624273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:42.624232    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:43.578239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:43.578199    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:43.735837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:43.735805    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:45.434928 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:45.434877    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:47.178702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:47.178664    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:47.341622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:47.341587    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:47.770545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:47.770517    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:49.535272 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:49.535237    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:52.077259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:52.077205    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:52.239196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:52.239157    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:52.404842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:52.404787    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:54.512411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.512367    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:54.718979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.718945    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"]
Oct 30 22:08:54.723140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.723116    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.830561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.830484    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"]
Oct 30 22:08:54.837604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.837574    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:54.857390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857362    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857398    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857422    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857563    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857608    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857715    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857770    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrpp\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-kube-api-access-tnrpp\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857852    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.857907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.857900    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.963763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.963714    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986192    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986284    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrpp\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-kube-api-access-tnrpp\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986347    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986394    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986477    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986515    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986541    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.986670    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.988352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.987650    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:54.988665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.988619    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.014685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:54.989759    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.014685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.011441    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.014945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.014908    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.015008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.014912    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.015008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.014942    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.050799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.050766    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrpp\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-kube-api-access-tnrpp\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.050943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.050770    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.337350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.337318    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:55.502237 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:55.502193    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice/crio-f0fe56e9de7feb62343783537ac8a17269449448a27d1ce96b64b7ae9f9e07cf\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0b508c_336f_486a_aad9_5b530c889cff.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:08:55.798012 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:08:55.797979    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08c39ab_2daf_4f0f_a2f6_51824ad30b1a.slice/crio-ec866f2dfaf2f946e61a32e856af00ccd0c7e487d5f98d68beb363b1ef20e167 WatchSource:0}: Error finding container ec866f2dfaf2f946e61a32e856af00ccd0c7e487d5f98d68beb363b1ef20e167: Status 404 returned error can't find the container with id ec866f2dfaf2f946e61a32e856af00ccd0c7e487d5f98d68beb363b1ef20e167
Oct 30 22:08:55.800301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.800266    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:08:55.800405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.800336    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:08:55.800405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.800366    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:08:55.801005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:55.800982    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"]
Oct 30 22:08:56.403222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:56.403186    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" event={"ID":"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a","Type":"ContainerStarted","Data":"5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46"}
Oct 30 22:08:56.403400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:56.403242    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" event={"ID":"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a","Type":"ContainerStarted","Data":"ec866f2dfaf2f946e61a32e856af00ccd0c7e487d5f98d68beb363b1ef20e167"}
Oct 30 22:08:56.547404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:56.547346    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podStartSLOduration=2.547331339 podStartE2EDuration="2.547331339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:56.544994807 +0000 UTC m=+1312.879571576" watchObservedRunningTime="2026-10-30 22:08:56.547331339 +0000 UTC m=+1312.881908107"
Oct 30 22:08:57.070838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:57.070800    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:08:57.341187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:57.341078    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:08:57.341670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:57.341606    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:08:57.341881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:57.341846    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:08:58.098554 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:08:58.098517    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:08:58.339661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:58.339514    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:08:58.339661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:58.339575    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:08:59.340668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:59.339681    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:08:59.340668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:08:59.339758    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:00.339995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:00.339949    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:00.340205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:00.340015    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:01.338660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:01.337961    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:01.338660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:01.338024    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:01.640562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:01.640467    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:02.338572 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:02.338536    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:09:02.338771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:02.338586    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:09:03.342729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:03.342698    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:09:03.343213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:03.343064    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:09:03.343631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:03.343610    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:09:04.120750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:04.120723    5503 scope.go:122] "RemoveContainer" containerID="014ccc8a7381cde8ec32b09f781f0d1f54dca896bd512a51f24ad34b68579459"
Oct 30 22:09:06.132456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.132421    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:06.258727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.258691    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"]
Oct 30 22:09:06.267129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.265329    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.312465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.312433    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"]
Oct 30 22:09:06.350958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.350924    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:06.357256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.357230    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-model-cache\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.357403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.357258    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2k25\" (UniqueName: \"kubernetes.io/projected/d07b7d4d-0016-477f-8edc-47db1175e93b-kube-api-access-f2k25\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.357403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.357377    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-home\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.357484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.357439    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-tmp-dir\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.357543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.357526    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07b7d4d-0016-477f-8edc-47db1175e93b-tls-certs\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.357584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.357554    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-dshm\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.360342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.360323    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b47e8d\"/\"distinct-w3-b47e8d-kserve-self-signed-certs\""
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.462703    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07b7d4d-0016-477f-8edc-47db1175e93b-tls-certs\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.462765    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-dshm\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.462906    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-model-cache\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.462933    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2k25\" (UniqueName: \"kubernetes.io/projected/d07b7d4d-0016-477f-8edc-47db1175e93b-kube-api-access-f2k25\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.463023    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-home\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.463101    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-tmp-dir\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.463599    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-tmp-dir\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.467187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.464280    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-model-cache\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.470206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.470178    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-dshm\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.472464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.470213    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-home\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.474338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.474301    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07b7d4d-0016-477f-8edc-47db1175e93b-tls-certs\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.544378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.544337    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2k25\" (UniqueName: \"kubernetes.io/projected/d07b7d4d-0016-477f-8edc-47db1175e93b-kube-api-access-f2k25\") pod \"distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:06.588802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:06.588758    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:07.014426 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:09:07.014400    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd07b7d4d_0016_477f_8edc_47db1175e93b.slice/crio-2248e5a872c54f630f9463944e820e7eee416be64b281b75f5c40fd57507d77a WatchSource:0}: Error finding container 2248e5a872c54f630f9463944e820e7eee416be64b281b75f5c40fd57507d77a: Status 404 returned error can't find the container with id 2248e5a872c54f630f9463944e820e7eee416be64b281b75f5c40fd57507d77a
Oct 30 22:09:07.015483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:07.015457    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"]
Oct 30 22:09:07.511777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:07.511742    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv" event={"ID":"d07b7d4d-0016-477f-8edc-47db1175e93b","Type":"ContainerStarted","Data":"775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed"}
Oct 30 22:09:07.511777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:07.511781    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv" event={"ID":"d07b7d4d-0016-477f-8edc-47db1175e93b","Type":"ContainerStarted","Data":"2248e5a872c54f630f9463944e820e7eee416be64b281b75f5c40fd57507d77a"}
Oct 30 22:09:07.589289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:07.589241    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:07.632885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:07.632830    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv" podStartSLOduration=1.6328143769999999 podStartE2EDuration="1.632814377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:07.630301196 +0000 UTC m=+1323.964877965" watchObservedRunningTime="2026-10-30 22:09:07.632814377 +0000 UTC m=+1323.967391146"
Oct 30 22:09:09.806720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:09.806690    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:10.600012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:10.599971    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:15.012297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:15.012254    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:09:15.198750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:15.198703    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.385198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:16.385162    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:16.589055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:16.589022    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:16.601443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:16.601418    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:16.789217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:16.789184    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:17.601479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:17.601453    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:09:19.948491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:19.948457    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:23.069401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:23.069361    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:24.629944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:24.629912    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:28.098922 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:09:28.098884    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:09:29.496062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:29.496023    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:31.542868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.542832    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"]
Oct 30 22:09:31.546867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.546848    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.603597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.603565    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:31.603802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.603688    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:09:31.603921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.603719    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:09:31.654516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.654482    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:09:31.657366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.657340    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"]
Oct 30 22:09:31.660282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.660262    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.660350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.660326    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.660489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.660470    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr66r\" (UniqueName: \"kubernetes.io/projected/104a5f93-9413-4823-b60b-7e9dab5232e2-kube-api-access-nr66r\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.761187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.761155    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.761451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.761235    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr66r\" (UniqueName: \"kubernetes.io/projected/104a5f93-9413-4823-b60b-7e9dab5232e2-kube-api-access-nr66r\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.761451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.761376    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.762049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.762027    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.763798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.763778    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.823696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.823616    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr66r\" (UniqueName: \"kubernetes.io/projected/104a5f93-9413-4823-b60b-7e9dab5232e2-kube-api-access-nr66r\") pod \"maas-api-revoke-sub-9adb60c4-75pj7\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:31.860724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:31.860697    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:32.236716 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:09:32.236680    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104a5f93_9413_4823_b60b_7e9dab5232e2.slice/crio-8214945228defbf22ff6ffa6a8f8c85338bdd4cd4485ee715c948abf15f98f06 WatchSource:0}: Error finding container 8214945228defbf22ff6ffa6a8f8c85338bdd4cd4485ee715c948abf15f98f06: Status 404 returned error can't find the container with id 8214945228defbf22ff6ffa6a8f8c85338bdd4cd4485ee715c948abf15f98f06
Oct 30 22:09:32.240014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.239689    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"]
Oct 30 22:09:32.542656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.542615    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"]
Oct 30 22:09:32.546567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.546549    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.605011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.604977    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"]
Oct 30 22:09:32.634260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.634222    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b47e8d\"/\"distinct-2-w3-b47e8d-kserve-self-signed-certs\""
Oct 30 22:09:32.674319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.674289    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-home\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.674319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.674323    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mj4x\" (UniqueName: \"kubernetes.io/projected/fc0dec83-8177-4bab-82dc-d62f1d5a7258-kube-api-access-8mj4x\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.674563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.674357    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tls-certs\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.674563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.674483    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-dshm\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.674563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.674516    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tmp-dir\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.674563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.674533    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-model-cache\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.695650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.695591    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7" event={"ID":"104a5f93-9413-4823-b60b-7e9dab5232e2","Type":"ContainerStarted","Data":"dadcaedd947283dafb4e0846b7d77ef84166ee16f0c11b4ac3308977ee3042ae"}
Oct 30 22:09:32.697594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.697562    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7" event={"ID":"104a5f93-9413-4823-b60b-7e9dab5232e2","Type":"ContainerStarted","Data":"8214945228defbf22ff6ffa6a8f8c85338bdd4cd4485ee715c948abf15f98f06"}
Oct 30 22:09:32.781703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.779545    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-home\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.781703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.779603    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mj4x\" (UniqueName: \"kubernetes.io/projected/fc0dec83-8177-4bab-82dc-d62f1d5a7258-kube-api-access-8mj4x\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.781703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.779693    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tls-certs\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.781703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.779760    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-dshm\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.781703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.779829    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tmp-dir\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.781703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.779853    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-model-cache\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.781703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.780901    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-home\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.795694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.782446    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tmp-dir\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.795921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.791411    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-model-cache\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.808677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.806343    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tls-certs\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.819231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.819198    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-dshm\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.833736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.833677    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7" podStartSLOduration=1.833658322 podStartE2EDuration="1.833658322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:32.830031527 +0000 UTC m=+1349.164608298" watchObservedRunningTime="2026-10-30 22:09:32.833658322 +0000 UTC m=+1349.168235091"
Oct 30 22:09:32.846738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.846709    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mj4x\" (UniqueName: \"kubernetes.io/projected/fc0dec83-8177-4bab-82dc-d62f1d5a7258-kube-api-access-8mj4x\") pod \"distinct-2-w3-b47e8d-kserve-d7c946555-cdc74\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") " pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:32.858584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:32.858557    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:33.436023 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:09:33.435990    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc0dec83_8177_4bab_82dc_d62f1d5a7258.slice/crio-254d4b7f8448de5ddca2c1b9e27262d4ce4f76e9a48e892b9e90feb8c884b247 WatchSource:0}: Error finding container 254d4b7f8448de5ddca2c1b9e27262d4ce4f76e9a48e892b9e90feb8c884b247: Status 404 returned error can't find the container with id 254d4b7f8448de5ddca2c1b9e27262d4ce4f76e9a48e892b9e90feb8c884b247
Oct 30 22:09:33.438934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.438914    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"]
Oct 30 22:09:33.704574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.704483    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" event={"ID":"fc0dec83-8177-4bab-82dc-d62f1d5a7258","Type":"ContainerStarted","Data":"24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717"}
Oct 30 22:09:33.704574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.704536    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" event={"ID":"fc0dec83-8177-4bab-82dc-d62f1d5a7258","Type":"ContainerStarted","Data":"254d4b7f8448de5ddca2c1b9e27262d4ce4f76e9a48e892b9e90feb8c884b247"}
Oct 30 22:09:33.706313 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.706288    5503 generic.go:356] "Generic (PLEG): container finished" podID="104a5f93-9413-4823-b60b-7e9dab5232e2" containerID="dadcaedd947283dafb4e0846b7d77ef84166ee16f0c11b4ac3308977ee3042ae" exitCode=0
Oct 30 22:09:33.706439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.706372    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7" event={"ID":"104a5f93-9413-4823-b60b-7e9dab5232e2","Type":"ContainerDied","Data":"dadcaedd947283dafb4e0846b7d77ef84166ee16f0c11b4ac3308977ee3042ae"}
Oct 30 22:09:33.749250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.748769    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:33.855661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.853178    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" podStartSLOduration=1.853156113 podStartE2EDuration="1.853156113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:33.846146242 +0000 UTC m=+1350.180723012" watchObservedRunningTime="2026-10-30 22:09:33.853156113 +0000 UTC m=+1350.187732882"
Oct 30 22:09:33.880689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.879140    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:33.901565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:33.901535    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:34.523591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.523556    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:34.862603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.862580    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:34.930860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.930826    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-xks-ca\") pod \"104a5f93-9413-4823-b60b-7e9dab5232e2\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") "
Oct 30 22:09:34.931054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.930871    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-openshift-ca\") pod \"104a5f93-9413-4823-b60b-7e9dab5232e2\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") "
Oct 30 22:09:34.931054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.931001    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104a5f93-9413-4823-b60b-7e9dab5232e2-kube-api-access-nr66r\" (UniqueName: \"kubernetes.io/projected/104a5f93-9413-4823-b60b-7e9dab5232e2-kube-api-access-nr66r\") pod \"104a5f93-9413-4823-b60b-7e9dab5232e2\" (UID: \"104a5f93-9413-4823-b60b-7e9dab5232e2\") "
Oct 30 22:09:34.931314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.931286    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-openshift-ca" pod "104a5f93-9413-4823-b60b-7e9dab5232e2" (UID: "104a5f93-9413-4823-b60b-7e9dab5232e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:34.931532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.931514    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.933086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.933053    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-xks-ca" pod "104a5f93-9413-4823-b60b-7e9dab5232e2" (UID: "104a5f93-9413-4823-b60b-7e9dab5232e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:34.933202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:34.933148    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104a5f93-9413-4823-b60b-7e9dab5232e2-kube-api-access-nr66r" pod "104a5f93-9413-4823-b60b-7e9dab5232e2" (UID: "104a5f93-9413-4823-b60b-7e9dab5232e2"). InnerVolumeSpecName "kube-api-access-nr66r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:35.032392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:35.032358    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104a5f93-9413-4823-b60b-7e9dab5232e2-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:35.032392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:35.032388    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr66r\" (UniqueName: \"kubernetes.io/projected/104a5f93-9413-4823-b60b-7e9dab5232e2-kube-api-access-nr66r\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:35.665251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:35.665211    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:35.719525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:35.719500    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"
Oct 30 22:09:35.719754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:35.719558    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7" event={"ID":"104a5f93-9413-4823-b60b-7e9dab5232e2","Type":"ContainerDied","Data":"8214945228defbf22ff6ffa6a8f8c85338bdd4cd4485ee715c948abf15f98f06"}
Oct 30 22:09:35.719754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:35.719600    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8214945228defbf22ff6ffa6a8f8c85338bdd4cd4485ee715c948abf15f98f06"
Oct 30 22:09:36.496299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:36.496262    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:37.047674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:37.041627    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:37.235525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:37.235480    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:39.413938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:39.413902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:41.718101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:41.718064    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:41.960907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:41.960874    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:42.858880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:42.858787    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:42.872020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:42.871995    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:43.781111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:43.781080    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:09:44.276366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:44.276327    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:44.936588 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:44.936550    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:45.149623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:45.149587    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:48.810545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:48.810510    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:51.408896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:51.408861    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:51.564301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:51.564267    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:53.546115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:53.546077    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:09:58.099358 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:09:58.099314    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:09:58.267325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:09:58.267290    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:02.844826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:02.844789    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:07.027582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:07.027541    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:10:07.513261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:07.512178    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:09.915858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:09.915803    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:10.753106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.753071    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"]
Oct 30 22:10:10.753781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.753757    5503 cpu_manager.go:414] "containerMap: removing container" podUID="104a5f93-9413-4823-b60b-7e9dab5232e2" containerName="revoke-keys"
Oct 30 22:10:10.753781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.753777    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104a5f93-9413-4823-b60b-7e9dab5232e2" containerName="revoke-keys"
Oct 30 22:10:10.754001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.753880    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="104a5f93-9413-4823-b60b-7e9dab5232e2" containerName="revoke-keys"
Oct 30 22:10:10.758337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.758320    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.815942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.815911    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-872730-openshift-default-dockercfg-9qtpz\""
Oct 30 22:10:10.851420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.851380    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-certs\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.851605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.851457    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-podinfo\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.851605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.851521    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvkh\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-kube-api-access-gqvkh\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.851754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.851617    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-envoy\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.851754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.851671    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-token\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.851863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.851826    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-socket\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.851963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.851946    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-credential-socket\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.852024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.852004    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc2c764-661b-4dbe-8bab-ddca18905cf3-istiod-ca-cert\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.852071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.852054    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-data\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.862665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.862587    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:10.867221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.867195    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"]
Oct 30 22:10:10.961712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.953559    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc2c764-661b-4dbe-8bab-ddca18905cf3-istiod-ca-cert\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.966399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.966371    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-data\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.967021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.967000    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-certs\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.967524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.967506    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-podinfo\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.968093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.968070    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvkh\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-kube-api-access-gqvkh\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.974657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.968534    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-envoy\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.975702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.975682    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-token\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.976016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.975951    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-socket\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.980903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.976256    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-credential-socket\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.981173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.967415    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-certs\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.981291 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.976350    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-socket\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.981386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.966892    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-data\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.981483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.976593    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-credential-socket\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.982035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.982013    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc2c764-661b-4dbe-8bab-ddca18905cf3-istiod-ca-cert\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:10.994987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:10.994922    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-envoy\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:11.001582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.001547    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-podinfo\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:11.033031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.032687    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvkh\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-kube-api-access-gqvkh\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:11.039131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.039043    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-token\") pod \"e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") " pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:11.077658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.077174    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:11.181706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.181016    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:11.549201 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:10:11.549170    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc2c764_661b_4dbe_8bab_ddca18905cf3.slice/crio-5930f5a12f2378ee90b0adaf681b79927a17325a7e37b547924ec62eec80ff08 WatchSource:0}: Error finding container 5930f5a12f2378ee90b0adaf681b79927a17325a7e37b547924ec62eec80ff08: Status 404 returned error can't find the container with id 5930f5a12f2378ee90b0adaf681b79927a17325a7e37b547924ec62eec80ff08
Oct 30 22:10:11.551675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.551652    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"]
Oct 30 22:10:11.551814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.551782    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:10:11.551860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.551848    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:10:11.551892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:11.551880    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:10:12.011098 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:12.011054    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" event={"ID":"abc2c764-661b-4dbe-8bab-ddca18905cf3","Type":"ContainerStarted","Data":"8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465"}
Oct 30 22:10:12.011488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:12.011137    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" event={"ID":"abc2c764-661b-4dbe-8bab-ddca18905cf3","Type":"ContainerStarted","Data":"5930f5a12f2378ee90b0adaf681b79927a17325a7e37b547924ec62eec80ff08"}
Oct 30 22:10:12.017798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:12.017768    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:12.080388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:12.080352    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:12.116371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:12.116333    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:12.141617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:12.141557    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" podStartSLOduration=2.141538545 podStartE2EDuration="2.141538545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:12.140607233 +0000 UTC m=+1388.475184002" watchObservedRunningTime="2026-10-30 22:10:12.141538545 +0000 UTC m=+1388.476115314"
Oct 30 22:10:13.018065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:13.018035    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:13.019068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:13.019045    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:10:14.944218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:14.944186    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:15.144673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:15.144630    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:15.306407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:15.306318    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:16.910387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:16.910347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:17.246266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:17.246167    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:18.287296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:18.287261    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:18.495864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:18.495828    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:21.679297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:21.679242    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:21.849103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:21.849070    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:22.005225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:22.005110    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:26.437105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:26.437059    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:28.098660 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:10:28.098610    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:10:31.039938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.039885    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:31.892925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.892891    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"]
Oct 30 22:10:31.896844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.896818    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:31.952924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.952892    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"]
Oct 30 22:10:31.991070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.991034    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-model-cache\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:31.991238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.991090    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-tmp-dir\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:31.991280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.991230    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa83512-318e-4a91-82cd-25a1404dabcd-tls-certs\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:31.991315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.991283    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cn4\" (UniqueName: \"kubernetes.io/projected/8fa83512-318e-4a91-82cd-25a1404dabcd-kube-api-access-47cn4\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:31.991399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.991376    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-dshm\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:31.991512 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.991466    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-home\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:31.994752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:31.994730    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:32.000220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.000203    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4cf21208\"/\"openshift-service-ca.crt\""
Oct 30 22:10:32.000303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.000227    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4cf21208\"/\"e2e-lifecycle-model-4cf21208-kserve-self-signed-certs\""
Oct 30 22:10:32.000303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.000227    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4cf21208\"/\"default-dockercfg-r79mq\""
Oct 30 22:10:32.000667 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.000654    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4cf21208\"/\"kube-root-ca.crt\""
Oct 30 22:10:32.093525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.093489    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-model-cache\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.094601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.094530    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-tmp-dir\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.094853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.094833    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa83512-318e-4a91-82cd-25a1404dabcd-tls-certs\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.094998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.094973    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-tmp-dir\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.095061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.094977    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cn4\" (UniqueName: \"kubernetes.io/projected/8fa83512-318e-4a91-82cd-25a1404dabcd-kube-api-access-47cn4\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.095113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.095068    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-dshm\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.095176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.095158    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-home\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.095816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.095497    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-home\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.095816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.094443    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-model-cache\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.114981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.114945    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa83512-318e-4a91-82cd-25a1404dabcd-tls-certs\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.137987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.137928    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-dshm\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.157930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.157895    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cn4\" (UniqueName: \"kubernetes.io/projected/8fa83512-318e-4a91-82cd-25a1404dabcd-kube-api-access-47cn4\") pod \"e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") " pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.209177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.209138    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:32.660532 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:10:32.660504    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa83512_318e_4a91_82cd_25a1404dabcd.slice/crio-1567d66bf8201f8255462452cb53cebbf9936911cd7a1d9eb0160af4b72864eb WatchSource:0}: Error finding container 1567d66bf8201f8255462452cb53cebbf9936911cd7a1d9eb0160af4b72864eb: Status 404 returned error can't find the container with id 1567d66bf8201f8255462452cb53cebbf9936911cd7a1d9eb0160af4b72864eb
Oct 30 22:10:32.663978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:32.663955    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"]
Oct 30 22:10:33.150323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:33.150290    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7" event={"ID":"8fa83512-318e-4a91-82cd-25a1404dabcd","Type":"ContainerStarted","Data":"d2a443169df79b9a084e5b8874ab890c87aabb29fa9a84bbe194e50ac28041b5"}
Oct 30 22:10:33.150809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:33.150333    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7" event={"ID":"8fa83512-318e-4a91-82cd-25a1404dabcd","Type":"ContainerStarted","Data":"1567d66bf8201f8255462452cb53cebbf9936911cd7a1d9eb0160af4b72864eb"}
Oct 30 22:10:33.209799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:33.209764    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:33.271785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:33.271726    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7" podStartSLOduration=2.271706898 podStartE2EDuration="2.271706898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.268857947 +0000 UTC m=+1409.603434716" watchObservedRunningTime="2026-10-30 22:10:33.271706898 +0000 UTC m=+1409.606283667"
Oct 30 22:10:35.563467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:35.563432    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:36.134137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.134100    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"]
Oct 30 22:10:36.138189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.138163    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.157202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.157172    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.157316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.157289    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwjn\" (UniqueName: \"kubernetes.io/projected/ba1da2c1-0f18-4f46-982b-8b85038125da-kube-api-access-xrwjn\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.157368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.157333    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.192225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.192189    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"]
Oct 30 22:10:36.240170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.240138    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:36.240170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.240167    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:10:36.240375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.240182    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:10:36.242881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.242867    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:10:36.258245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.258221    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.258387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.258295    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwjn\" (UniqueName: \"kubernetes.io/projected/ba1da2c1-0f18-4f46-982b-8b85038125da-kube-api-access-xrwjn\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.258387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.258323    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.259050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.259028    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.260579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.260555    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.321731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.321681    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwjn\" (UniqueName: \"kubernetes.io/projected/ba1da2c1-0f18-4f46-982b-8b85038125da-kube-api-access-xrwjn\") pod \"maas-api-revoke-sub-180aaba5-rzqmf\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.452308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.452278    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:36.821677 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:10:36.821618    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1da2c1_0f18_4f46_982b_8b85038125da.slice/crio-50a144e6aefae1b5e7846efd112477188314e969b91f8b029c3c6dc7e8557194 WatchSource:0}: Error finding container 50a144e6aefae1b5e7846efd112477188314e969b91f8b029c3c6dc7e8557194: Status 404 returned error can't find the container with id 50a144e6aefae1b5e7846efd112477188314e969b91f8b029c3c6dc7e8557194
Oct 30 22:10:36.824565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:36.824501    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"]
Oct 30 22:10:37.181498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:37.181462    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf" event={"ID":"ba1da2c1-0f18-4f46-982b-8b85038125da","Type":"ContainerStarted","Data":"4d7a51a89b6887350c56c4bac8b13c4bd0fce1c5380698e2fec0e6108a254371"}
Oct 30 22:10:37.181699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:37.181538    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf" event={"ID":"ba1da2c1-0f18-4f46-982b-8b85038125da","Type":"ContainerStarted","Data":"50a144e6aefae1b5e7846efd112477188314e969b91f8b029c3c6dc7e8557194"}
Oct 30 22:10:37.302026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:37.301973    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf" podStartSLOduration=1.301959249 podStartE2EDuration="1.301959249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:37.299695138 +0000 UTC m=+1413.634271906" watchObservedRunningTime="2026-10-30 22:10:37.301959249 +0000 UTC m=+1413.636536018"
Oct 30 22:10:38.188489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:38.188453    5503 generic.go:356] "Generic (PLEG): container finished" podID="ba1da2c1-0f18-4f46-982b-8b85038125da" containerID="4d7a51a89b6887350c56c4bac8b13c4bd0fce1c5380698e2fec0e6108a254371" exitCode=0
Oct 30 22:10:38.188906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:38.188501    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf" event={"ID":"ba1da2c1-0f18-4f46-982b-8b85038125da","Type":"ContainerDied","Data":"4d7a51a89b6887350c56c4bac8b13c4bd0fce1c5380698e2fec0e6108a254371"}
Oct 30 22:10:39.358669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.358630    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:39.397131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.397097    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-openshift-ca\") pod \"ba1da2c1-0f18-4f46-982b-8b85038125da\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") "
Oct 30 22:10:39.397305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.397149    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-xks-ca\") pod \"ba1da2c1-0f18-4f46-982b-8b85038125da\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") "
Oct 30 22:10:39.397305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.397184    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba1da2c1-0f18-4f46-982b-8b85038125da-kube-api-access-xrwjn\" (UniqueName: \"kubernetes.io/projected/ba1da2c1-0f18-4f46-982b-8b85038125da-kube-api-access-xrwjn\") pod \"ba1da2c1-0f18-4f46-982b-8b85038125da\" (UID: \"ba1da2c1-0f18-4f46-982b-8b85038125da\") "
Oct 30 22:10:39.397506 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.397480    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-openshift-ca" pod "ba1da2c1-0f18-4f46-982b-8b85038125da" (UID: "ba1da2c1-0f18-4f46-982b-8b85038125da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:39.399272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.399245    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-xks-ca" pod "ba1da2c1-0f18-4f46-982b-8b85038125da" (UID: "ba1da2c1-0f18-4f46-982b-8b85038125da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:39.399363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.399304    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1da2c1-0f18-4f46-982b-8b85038125da-kube-api-access-xrwjn" pod "ba1da2c1-0f18-4f46-982b-8b85038125da" (UID: "ba1da2c1-0f18-4f46-982b-8b85038125da"). InnerVolumeSpecName "kube-api-access-xrwjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:39.499215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.499132    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwjn\" (UniqueName: \"kubernetes.io/projected/ba1da2c1-0f18-4f46-982b-8b85038125da-kube-api-access-xrwjn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:39.499215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.499171    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:39.499215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:39.499188    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba1da2c1-0f18-4f46-982b-8b85038125da-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:40.202521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:40.202476    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:40.203467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:40.203449    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"
Oct 30 22:10:40.203557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:40.203489    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf" event={"ID":"ba1da2c1-0f18-4f46-982b-8b85038125da","Type":"ContainerDied","Data":"50a144e6aefae1b5e7846efd112477188314e969b91f8b029c3c6dc7e8557194"}
Oct 30 22:10:40.203557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:40.203545    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a144e6aefae1b5e7846efd112477188314e969b91f8b029c3c6dc7e8557194"
Oct 30 22:10:42.209352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:42.209300    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:42.223163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:42.223134    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:43.220716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:43.220688    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:10:44.320593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:44.320554    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:10:44.971838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:44.971799    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:46.861683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:46.861631    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:47.515902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.515868    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"]
Oct 30 22:10:47.516365 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.516351    5503 cpu_manager.go:414] "containerMap: removing container" podUID="ba1da2c1-0f18-4f46-982b-8b85038125da" containerName="revoke-keys"
Oct 30 22:10:47.516425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.516366    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba1da2c1-0f18-4f46-982b-8b85038125da" containerName="revoke-keys"
Oct 30 22:10:47.516482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.516474    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba1da2c1-0f18-4f46-982b-8b85038125da" containerName="revoke-keys"
Oct 30 22:10:47.520886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.520866    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.661659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.658369    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-161111-openshift-default-dockercfg-dn2ch\""
Oct 30 22:10:47.702660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.701604    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"]
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.709444    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b132c2-4393-49be-9adb-dd50b45d2322-istiod-ca-cert\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.709543    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-token\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.709678    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6db7\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-kube-api-access-k6db7\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.709777    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-envoy\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.709837    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-credential-socket\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.709905    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-socket\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.709977    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-certs\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.710025    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-data\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.713700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.710096    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-podinfo\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.798706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.696609    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:47.820459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.820414    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-certs\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.824004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.823971    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-data\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.824744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.824720    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-podinfo\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.825503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.825482    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b132c2-4393-49be-9adb-dd50b45d2322-istiod-ca-cert\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.825845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.825821    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-token\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.826059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.826037    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6db7\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-kube-api-access-k6db7\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.826257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.826239    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-envoy\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.826402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.826383    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-credential-socket\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.826551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.826536    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-socket\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.826798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.821166    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-certs\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.826929 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.826315    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b132c2-4393-49be-9adb-dd50b45d2322-istiod-ca-cert\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.827290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.827270    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-credential-socket\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.827667 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.827629    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-socket\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.827854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.824604    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-data\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.835554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.835527    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-envoy\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.876555 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.876520    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-podinfo\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:47.957820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.957603    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:47.968671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:47.965293    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-token\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:48.001395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:48.001358    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6db7\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-kube-api-access-k6db7\") pod \"e2e-api-a-161111-openshift-default-cf8877bd7-tq97q\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") " pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:48.153704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:48.153670    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:48.594740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:48.594697    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"]
Oct 30 22:10:48.596217 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:10:48.596192    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b132c2_4393_49be_9adb_dd50b45d2322.slice/crio-b709c96c9f83f221048639d7c7ce5ef0afb41d9fb35a983a6e7c95cc6256e43a WatchSource:0}: Error finding container b709c96c9f83f221048639d7c7ce5ef0afb41d9fb35a983a6e7c95cc6256e43a: Status 404 returned error can't find the container with id b709c96c9f83f221048639d7c7ce5ef0afb41d9fb35a983a6e7c95cc6256e43a
Oct 30 22:10:48.598794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:48.598763    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:10:48.598858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:48.598831    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:10:48.598900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:48.598860    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:10:49.291330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:49.291292    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" event={"ID":"e2b132c2-4393-49be-9adb-dd50b45d2322","Type":"ContainerStarted","Data":"fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b"}
Oct 30 22:10:49.291741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:49.291391    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" event={"ID":"e2b132c2-4393-49be-9adb-dd50b45d2322","Type":"ContainerStarted","Data":"b709c96c9f83f221048639d7c7ce5ef0afb41d9fb35a983a6e7c95cc6256e43a"}
Oct 30 22:10:49.414939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:49.414888    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podStartSLOduration=2.414875663 podStartE2EDuration="2.414875663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:49.413795702 +0000 UTC m=+1425.748372471" watchObservedRunningTime="2026-10-30 22:10:49.414875663 +0000 UTC m=+1425.749452431"
Oct 30 22:10:49.796442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:49.796342    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:50.154236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:50.154210    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:50.160752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:50.160725    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:50.297327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:50.297297    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:50.298083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:50.298065    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:10:52.305396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:52.305361    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:52.492977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:52.492941    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:52.780013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:52.779984    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:54.665322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:54.665288    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:55.759705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:55.757218    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:55.953656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:55.943241    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:58.099134 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:10:58.099097    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:10:59.676236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:59.676199    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:10:59.908660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:10:59.908205    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:00.065522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:00.065447    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:04.618124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:04.618071    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:05.160425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.160389    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"]
Oct 30 22:11:05.164269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.164215    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.247745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.247702    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"]
Oct 30 22:11:05.250187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250157    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.250305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250228    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.250400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250372    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.250460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250436    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg928\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-kube-api-access-wg928\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.250517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250485    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.250574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250547    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.250695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250675    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.250865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250841    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.251011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.250928    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.351877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.351850    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.351898    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.351928    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.351964    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.351989    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352038    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352074    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg928\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-kube-api-access-wg928\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352104    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352159    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352283    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352368    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352418    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352487    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.352875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.352853    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.354674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.354649    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.354782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.354764    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.415357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.415295    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg928\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-kube-api-access-wg928\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.415357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.415332    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.477347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.477323    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:05.896776 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:11:05.896746    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b7fd88_f60c_4e68_9de5_6180cf7ffc91.slice/crio-97dae7c40af1c1b454815112f51a2808ff16c2c6639887506cd29925acab43d5 WatchSource:0}: Error finding container 97dae7c40af1c1b454815112f51a2808ff16c2c6639887506cd29925acab43d5: Status 404 returned error can't find the container with id 97dae7c40af1c1b454815112f51a2808ff16c2c6639887506cd29925acab43d5
Oct 30 22:11:05.899622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.899586    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:11:05.899733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.899683    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:11:05.899733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.899701    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"]
Oct 30 22:11:05.899733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:05.899714    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:11:06.434673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:06.434627    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" event={"ID":"27b7fd88-f60c-4e68-9de5-6180cf7ffc91","Type":"ContainerStarted","Data":"7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380"}
Oct 30 22:11:06.434878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:06.434693    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" event={"ID":"27b7fd88-f60c-4e68-9de5-6180cf7ffc91","Type":"ContainerStarted","Data":"97dae7c40af1c1b454815112f51a2808ff16c2c6639887506cd29925acab43d5"}
Oct 30 22:11:06.477918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:06.477887    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:06.479314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:06.479288    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:06.479463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:06.479339    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:06.559678 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:06.559611    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podStartSLOduration=1.559597649 podStartE2EDuration="1.559597649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:06.557701184 +0000 UTC m=+1442.892277954" watchObservedRunningTime="2026-10-30 22:11:06.559597649 +0000 UTC m=+1442.894174465"
Oct 30 22:11:07.477965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:07.477923    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:07.478369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:07.477993    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:08.477872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:08.477835    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:08.478053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:08.477887    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:09.273248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:09.273208    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:09.478510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:09.478474    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:09.478921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:09.478522    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:10.478173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:10.478139    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:10.478332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:10.478194    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:11.477805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:11.477773    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:11.478287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:11.477831    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:12.478457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:12.478401    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:12.478891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:12.478466    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:13.485661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:13.479481    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:13.485661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:13.479541    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:14.287773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:14.287740    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:14.781727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:14.781441    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:14.783645 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:14.783570    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:14.785938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:14.785915    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:11:16.322956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.321730    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.392267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.391845    5503 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.427664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.426715    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.613815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.613721    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv" podUID="d07b7d4d-0016-477f-8edc-47db1175e93b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.614330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.614300    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:11:16.649788 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.649735    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.781182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.781140    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" podUID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.806779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.806723    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.862914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.862864    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:16.934773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:16.934724    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:17.283833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:17.283730    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:17.436659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:17.432721    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:17.436659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:17.433237    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:11:17.511457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:17.511427    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:11:17.684662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:17.684600    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:19.226556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:19.226495    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:19.812459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:19.812426    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:22.229541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.229505    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"]
Oct 30 22:11:22.233288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.233267    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.288890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.288857    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"]
Oct 30 22:11:22.336353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.336319    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:22.336353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.336325    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:11:22.336544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.336372    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:11:22.340707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.340685    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:11:22.407228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.407185    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.407435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.407411    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.407506 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.407489    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5vl\" (UniqueName: \"kubernetes.io/projected/39cbb7a3-59bb-4473-895e-71a1e3f11665-kube-api-access-kn5vl\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.509095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.509017    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.509259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.509142    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.509259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.509189    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5vl\" (UniqueName: \"kubernetes.io/projected/39cbb7a3-59bb-4473-895e-71a1e3f11665-kube-api-access-kn5vl\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.509906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.509883    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.511471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.511437    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.571622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.571588    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5vl\" (UniqueName: \"kubernetes.io/projected/39cbb7a3-59bb-4473-895e-71a1e3f11665-kube-api-access-kn5vl\") pod \"maas-api-revoke-keys-e2e-api-a-d5e5df-zl556\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:22.850230 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:22.850145    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:23.224817 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:11:23.224786    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cbb7a3_59bb_4473_895e_71a1e3f11665.slice/crio-2ff65f1a98e0aeca0f99e6e335b0680e835117de82cc95ceb79758c5ff4ae5e7 WatchSource:0}: Error finding container 2ff65f1a98e0aeca0f99e6e335b0680e835117de82cc95ceb79758c5ff4ae5e7: Status 404 returned error can't find the container with id 2ff65f1a98e0aeca0f99e6e335b0680e835117de82cc95ceb79758c5ff4ae5e7
Oct 30 22:11:23.227521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.227501    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"]
Oct 30 22:11:23.630068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.630038    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerID="8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465" exitCode=0
Oct 30 22:11:23.630607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.630121    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" event={"ID":"abc2c764-661b-4dbe-8bab-ddca18905cf3","Type":"ContainerDied","Data":"8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465"}
Oct 30 22:11:23.630607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.630598    5503 scope.go:122] "RemoveContainer" containerID="8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465"
Oct 30 22:11:23.631988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.631951    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556" event={"ID":"39cbb7a3-59bb-4473-895e-71a1e3f11665","Type":"ContainerStarted","Data":"937db330c52004036f59f493e3f08be032309bbb12d9255d698ff5a0f92f21e0"}
Oct 30 22:11:23.632126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.632007    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556" event={"ID":"39cbb7a3-59bb-4473-895e-71a1e3f11665","Type":"ContainerStarted","Data":"2ff65f1a98e0aeca0f99e6e335b0680e835117de82cc95ceb79758c5ff4ae5e7"}
Oct 30 22:11:23.632334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.632298    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:11:23.632407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.632390    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:11:23.632461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.632437    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:11:23.883200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:23.883094    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556" podStartSLOduration=1.883077605 podStartE2EDuration="1.883077605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:23.879693642 +0000 UTC m=+1460.214270415" watchObservedRunningTime="2026-10-30 22:11:23.883077605 +0000 UTC m=+1460.217654374"
Oct 30 22:11:24.647351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:24.647271    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" event={"ID":"abc2c764-661b-4dbe-8bab-ddca18905cf3","Type":"ContainerStarted","Data":"6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd"}
Oct 30 22:11:24.649098 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:24.649062    5503 generic.go:356] "Generic (PLEG): container finished" podID="39cbb7a3-59bb-4473-895e-71a1e3f11665" containerID="937db330c52004036f59f493e3f08be032309bbb12d9255d698ff5a0f92f21e0" exitCode=0
Oct 30 22:11:24.649219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:24.649134    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556" event={"ID":"39cbb7a3-59bb-4473-895e-71a1e3f11665","Type":"ContainerDied","Data":"937db330c52004036f59f493e3f08be032309bbb12d9255d698ff5a0f92f21e0"}
Oct 30 22:11:24.946568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:24.946530    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:25.078208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.077884    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:11:25.079348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.079312    5503 patch_prober.go:29] interesting pod/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:11:25.079513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.079372    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:11:25.844546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.844511    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:25.874690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.874659    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39cbb7a3-59bb-4473-895e-71a1e3f11665-kube-api-access-kn5vl\" (UniqueName: \"kubernetes.io/projected/39cbb7a3-59bb-4473-895e-71a1e3f11665-kube-api-access-kn5vl\") pod \"39cbb7a3-59bb-4473-895e-71a1e3f11665\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") "
Oct 30 22:11:25.874884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.874866    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-xks-ca\") pod \"39cbb7a3-59bb-4473-895e-71a1e3f11665\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") "
Oct 30 22:11:25.874953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.874937    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-openshift-ca\") pod \"39cbb7a3-59bb-4473-895e-71a1e3f11665\" (UID: \"39cbb7a3-59bb-4473-895e-71a1e3f11665\") "
Oct 30 22:11:25.875244 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.875223    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-openshift-ca" pod "39cbb7a3-59bb-4473-895e-71a1e3f11665" (UID: "39cbb7a3-59bb-4473-895e-71a1e3f11665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:25.875406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.875388    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:25.876691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.876669    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cbb7a3-59bb-4473-895e-71a1e3f11665-kube-api-access-kn5vl" pod "39cbb7a3-59bb-4473-895e-71a1e3f11665" (UID: "39cbb7a3-59bb-4473-895e-71a1e3f11665"). InnerVolumeSpecName "kube-api-access-kn5vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:25.876824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.876804    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-xks-ca" pod "39cbb7a3-59bb-4473-895e-71a1e3f11665" (UID: "39cbb7a3-59bb-4473-895e-71a1e3f11665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:25.978178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.978088    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39cbb7a3-59bb-4473-895e-71a1e3f11665-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:25.978178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:25.978140    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn5vl\" (UniqueName: \"kubernetes.io/projected/39cbb7a3-59bb-4473-895e-71a1e3f11665-kube-api-access-kn5vl\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:26.078569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:26.078535    5503 patch_prober.go:29] interesting pod/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:11:26.078782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:26.078595    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:11:26.663535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:26.663509    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"
Oct 30 22:11:26.663746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:26.663545    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556" event={"ID":"39cbb7a3-59bb-4473-895e-71a1e3f11665","Type":"ContainerDied","Data":"2ff65f1a98e0aeca0f99e6e335b0680e835117de82cc95ceb79758c5ff4ae5e7"}
Oct 30 22:11:26.663746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:26.663626    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff65f1a98e0aeca0f99e6e335b0680e835117de82cc95ceb79758c5ff4ae5e7"
Oct 30 22:11:27.078214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:27.078133    5503 patch_prober.go:29] interesting pod/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:11:27.078214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:27.078200    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:11:28.026663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:28.019735    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:11:28.078434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:28.078397    5503 patch_prober.go:29] interesting pod/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:11:28.078820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:28.078469    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:11:28.100360 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:11:28.100319    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:11:29.078123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:29.078085    5503 patch_prober.go:29] interesting pod/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:11:29.078308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:29.078141    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:11:29.525358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:29.525303    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:30.083018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:30.082986    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:11:30.084147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:30.084126    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:11:34.122050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:34.122012    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:34.308124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:34.307879    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:34.464020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:34.463990    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:38.675665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:38.675609    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:38.845897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:38.845862    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:39.016495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:39.016379    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:43.054456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:43.054416    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"]
Oct 30 22:11:43.104858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:43.104826    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d2de1-4dj79"]
Oct 30 22:11:43.438841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:43.438802    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:43.796249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:43.796162    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c1f5e8-504a-4764-a7db-fdfe9444c58f" path="/var/lib/kubelet/pods/83c1f5e8-504a-4764-a7db-fdfe9444c58f/volumes"
Oct 30 22:11:47.933320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:47.933281    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:52.561987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:52.561945    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:55.650172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:55.650120    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:11:57.358320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:57.358285    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:57.636715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:57.636472    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.098269 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:11:58.098235    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:11:58.141187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:58.141153    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:58.971095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:58.971002    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:11:59.330712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:11:59.323795    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:00.056576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:00.056534    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"]
Oct 30 22:12:00.106905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:00.106874    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8256e71c-ldkd8"]
Oct 30 22:12:01.789420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:01.789389    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275c7d4a-5564-433c-afb1-07f38c142238" path="/var/lib/kubelet/pods/275c7d4a-5564-433c-afb1-07f38c142238/volumes"
Oct 30 22:12:02.367242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:02.367205    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:02.977149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:02.977065    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:03.154962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:03.154931    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:03.309905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:03.309824    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:03.974682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:03.974652    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:12:03.982388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:03.982367    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:12:05.997599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:05.996887    5503 patch_prober.go:29] interesting pod/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": read tcp 10.134.0.2:37450->10.134.0.76:15021: read: connection reset by peer" start-of-body=
Oct 30 22:12:05.997599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:05.996959    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": read tcp 10.134.0.2:37450->10.134.0.76:15021: read: connection reset by peer"
Oct 30 22:12:06.143722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.143687    5503 generic.go:356] "Generic (PLEG): container finished" podID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerID="fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b" exitCode=0
Oct 30 22:12:06.143924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.143761    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" event={"ID":"e2b132c2-4393-49be-9adb-dd50b45d2322","Type":"ContainerDied","Data":"fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b"}
Oct 30 22:12:06.144236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.144218    5503 scope.go:122] "RemoveContainer" containerID="fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b"
Oct 30 22:12:06.145802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.145767    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:12:06.145904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.145836    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:12:06.145904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.145869    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:12:06.224510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.224456    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:06.412677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:06.412593    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:07.154055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.154021    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" event={"ID":"e2b132c2-4393-49be-9adb-dd50b45d2322","Type":"ContainerStarted","Data":"130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54"}
Oct 30 22:12:07.341825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.341788    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"]
Oct 30 22:12:07.342614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.342593    5503 cpu_manager.go:414] "containerMap: removing container" podUID="39cbb7a3-59bb-4473-895e-71a1e3f11665" containerName="revoke-keys"
Oct 30 22:12:07.342685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.342618    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39cbb7a3-59bb-4473-895e-71a1e3f11665" containerName="revoke-keys"
Oct 30 22:12:07.342805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.342792    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="39cbb7a3-59bb-4473-895e-71a1e3f11665" containerName="revoke-keys"
Oct 30 22:12:07.345513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.345490    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.388410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.388377    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"]
Oct 30 22:12:07.399458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.399426    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:07.399664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.399506    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:07.399664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.399523    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:07.399664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.399580    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:12:07.441282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.441145    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:07.473839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.473784    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bmcw\" (UniqueName: \"kubernetes.io/projected/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-kube-api-access-8bmcw\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.474229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.474203    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.474524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.474504    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.575478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.575446    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bmcw\" (UniqueName: \"kubernetes.io/projected/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-kube-api-access-8bmcw\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.575689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.575597    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.575797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.575771    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.576520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.576496    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.579532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.578053    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.642136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.642100    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bmcw\" (UniqueName: \"kubernetes.io/projected/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-kube-api-access-8bmcw\") pod \"maas-api-revoke-sub-52f0c9fa-bjrjm\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:07.658290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:07.658263    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:08.041562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:08.041494    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"]
Oct 30 22:12:08.044455 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:12:08.044428    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c5bbfb_537d_4d41_a2c7_b4fab648f74b.slice/crio-4b656da49160124739f543e08fc4c2648ee575f0d9afee5a613bbd9d3bece126 WatchSource:0}: Error finding container 4b656da49160124739f543e08fc4c2648ee575f0d9afee5a613bbd9d3bece126: Status 404 returned error can't find the container with id 4b656da49160124739f543e08fc4c2648ee575f0d9afee5a613bbd9d3bece126
Oct 30 22:12:08.154687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:08.154646    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:12:08.155139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:08.154865    5503 patch_prober.go:29] interesting pod/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:12:08.155139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:08.154930    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:12:08.161991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:08.161961    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm" event={"ID":"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b","Type":"ContainerStarted","Data":"4b656da49160124739f543e08fc4c2648ee575f0d9afee5a613bbd9d3bece126"}
Oct 30 22:12:09.154891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:09.154856    5503 patch_prober.go:29] interesting pod/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:12:09.155327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:09.154908    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:12:09.170711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:09.170676    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm" event={"ID":"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b","Type":"ContainerStarted","Data":"c9f635e299f9824c103a2d50f537f1b9259b7fdb9ef48a35faefcec9e2eafa7b"}
Oct 30 22:12:09.301359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:09.301293    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm" podStartSLOduration=2.301275577 podStartE2EDuration="2.301275577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:09.299781513 +0000 UTC m=+1505.634358282" watchObservedRunningTime="2026-10-30 22:12:09.301275577 +0000 UTC m=+1505.635852346"
Oct 30 22:12:10.106690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.106657    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7700-43fbd6"]
Oct 30 22:12:10.109324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.109309    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:10.154923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.154890    5503 patch_prober.go:29] interesting pod/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:12:10.155346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.154948    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:12:10.162966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.162934    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-43fbd6"]
Oct 30 22:12:10.178874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.178832    5503 generic.go:356] "Generic (PLEG): container finished" podID="d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" containerID="c9f635e299f9824c103a2d50f537f1b9259b7fdb9ef48a35faefcec9e2eafa7b" exitCode=0
Oct 30 22:12:10.179028 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.178912    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm" event={"ID":"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b","Type":"ContainerDied","Data":"c9f635e299f9824c103a2d50f537f1b9259b7fdb9ef48a35faefcec9e2eafa7b"}
Oct 30 22:12:10.208919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.208896    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmbvp\""
Oct 30 22:12:10.260288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.260256    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqjb\" (UniqueName: \"kubernetes.io/projected/7401983b-b5eb-4ac6-9bd8-c02600f955c0-kube-api-access-2tqjb\") pod \"test-curl-7700-43fbd6\" (UID: \"7401983b-b5eb-4ac6-9bd8-c02600f955c0\") " pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:10.361656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.361553    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqjb\" (UniqueName: \"kubernetes.io/projected/7401983b-b5eb-4ac6-9bd8-c02600f955c0-kube-api-access-2tqjb\") pod \"test-curl-7700-43fbd6\" (UID: \"7401983b-b5eb-4ac6-9bd8-c02600f955c0\") " pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:10.440475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.440441    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqjb\" (UniqueName: \"kubernetes.io/projected/7401983b-b5eb-4ac6-9bd8-c02600f955c0-kube-api-access-2tqjb\") pod \"test-curl-7700-43fbd6\" (UID: \"7401983b-b5eb-4ac6-9bd8-c02600f955c0\") " pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:10.444360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.444327    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:10.811881 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:12:10.811853    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7401983b_b5eb_4ac6_9bd8_c02600f955c0.slice/crio-d8aa8e8ab247f9965618cf303577139bb5393e38acad87cf8864df2d9fe914af WatchSource:0}: Error finding container d8aa8e8ab247f9965618cf303577139bb5393e38acad87cf8864df2d9fe914af: Status 404 returned error can't find the container with id d8aa8e8ab247f9965618cf303577139bb5393e38acad87cf8864df2d9fe914af
Oct 30 22:12:10.814411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.814390    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7700-43fbd6"]
Oct 30 22:12:10.814483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:10.814458    5503 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:12:11.154181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.154151    5503 patch_prober.go:29] interesting pod/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:12:11.154355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.154209    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:12:11.186162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.186125    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-43fbd6" event={"ID":"7401983b-b5eb-4ac6-9bd8-c02600f955c0","Type":"ContainerStarted","Data":"d8aa8e8ab247f9965618cf303577139bb5393e38acad87cf8864df2d9fe914af"}
Oct 30 22:12:11.378186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.378161    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:11.480244 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.480159    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-xks-ca\") pod \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") "
Oct 30 22:12:11.480403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.480255    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-openshift-ca\") pod \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") "
Oct 30 22:12:11.480454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.480436    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-kube-api-access-8bmcw\" (UniqueName: \"kubernetes.io/projected/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-kube-api-access-8bmcw\") pod \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\" (UID: \"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b\") "
Oct 30 22:12:11.480600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.480574    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-openshift-ca" pod "d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" (UID: "d5c5bbfb-537d-4d41-a2c7-b4fab648f74b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:11.480813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.480798    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:11.482285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.482263    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-xks-ca" pod "d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" (UID: "d5c5bbfb-537d-4d41-a2c7-b4fab648f74b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:11.482520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.482502    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-kube-api-access-8bmcw" pod "d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" (UID: "d5c5bbfb-537d-4d41-a2c7-b4fab648f74b"). InnerVolumeSpecName "kube-api-access-8bmcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:11.581946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.581911    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bmcw\" (UniqueName: \"kubernetes.io/projected/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-kube-api-access-8bmcw\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:11.581946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.581942    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:11.950781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:11.950744    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:12.155267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:12.155234    5503 patch_prober.go:29] interesting pod/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:12:12.155395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:12.155308    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:12:12.194196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:12.194132    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"
Oct 30 22:12:12.194568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:12.194243    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm" event={"ID":"d5c5bbfb-537d-4d41-a2c7-b4fab648f74b","Type":"ContainerDied","Data":"4b656da49160124739f543e08fc4c2648ee575f0d9afee5a613bbd9d3bece126"}
Oct 30 22:12:12.194568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:12.194333    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b656da49160124739f543e08fc4c2648ee575f0d9afee5a613bbd9d3bece126"
Oct 30 22:12:13.158096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:13.158067    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:12:13.158581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:13.158556    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:12:13.159153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:13.159137    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:12:13.201177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:13.201144    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7700-43fbd6" event={"ID":"7401983b-b5eb-4ac6-9bd8-c02600f955c0","Type":"ContainerStarted","Data":"49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f"}
Oct 30 22:12:15.071520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:15.071462    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7700-43fbd6" podStartSLOduration=3.761308118 podStartE2EDuration="5.071444051s" totalImagesPullingTime="1.310135933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:13.575838198 +0000 UTC m=+1509.910414967" watchObservedRunningTime="2026-10-30 22:12:15.071444051 +0000 UTC m=+1511.406020823"
Oct 30 22:12:15.073990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:15.073960    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7700-43fbd6"]
Oct 30 22:12:15.078544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:15.078517    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7700-43fbd6"]
Oct 30 22:12:15.213505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:15.213466    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7700-43fbd6" podUID="7401983b-b5eb-4ac6-9bd8-c02600f955c0" containerName="test-curl-7700-43fbd6" containerID="cri-o://49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f" gracePeriod=2
Oct 30 22:12:15.266048 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:12:15.266012    5503 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7700-43fbd6\" is forbidden: User \"system:node:ip-10-0-141-86.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-86.us-west-2.compute.internal' and this object" podUID="7401983b-b5eb-4ac6-9bd8-c02600f955c0" pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:16.017855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.017812    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"]
Oct 30 22:12:16.018998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.018969    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7401983b-b5eb-4ac6-9bd8-c02600f955c0" containerName="test-curl-7700-43fbd6"
Oct 30 22:12:16.019153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.019138    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7401983b-b5eb-4ac6-9bd8-c02600f955c0" containerName="test-curl-7700-43fbd6"
Oct 30 22:12:16.019262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.019252    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" containerName="revoke-keys"
Oct 30 22:12:16.019335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.019325    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" containerName="revoke-keys"
Oct 30 22:12:16.019721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.019703    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7401983b-b5eb-4ac6-9bd8-c02600f955c0" containerName="test-curl-7700-43fbd6"
Oct 30 22:12:16.019838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.019828    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" containerName="revoke-keys"
Oct 30 22:12:16.023704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.023682    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.042235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.042211    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhld\" (UniqueName: \"kubernetes.io/projected/d1500251-e98d-4183-8b38-dede0b91244e-kube-api-access-wvhld\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.042385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.042363    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.042422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.042409    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.075783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.075753    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"]
Oct 30 22:12:16.123681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.123627    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:16.123681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.123656    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:16.123899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.123665    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:16.128481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.128460    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:12:16.143238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.143210    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.143317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.143258    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.143424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.143408    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhld\" (UniqueName: \"kubernetes.io/projected/d1500251-e98d-4183-8b38-dede0b91244e-kube-api-access-wvhld\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.144037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.144019    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.145659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.145625    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.206074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.206042    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhld\" (UniqueName: \"kubernetes.io/projected/d1500251-e98d-4183-8b38-dede0b91244e-kube-api-access-wvhld\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.336163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.336075    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:16.679702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.670256    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:16.745246 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:12:16.744844    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1500251_e98d_4183_8b38_dede0b91244e.slice/crio-d562006e66f04a8997a398c5579a126dd2d6c05085089c31abd431ab63c6e309 WatchSource:0}: Error finding container d562006e66f04a8997a398c5579a126dd2d6c05085089c31abd431ab63c6e309: Status 404 returned error can't find the container with id d562006e66f04a8997a398c5579a126dd2d6c05085089c31abd431ab63c6e309
Oct 30 22:12:16.751256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:16.751221    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"]
Oct 30 22:12:17.229735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.229698    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd" event={"ID":"d1500251-e98d-4183-8b38-dede0b91244e","Type":"ContainerStarted","Data":"dac4a2349e0abdced608e59259b4c9208257807d780fdb40e57fc7fec8342427"}
Oct 30 22:12:17.230083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.229755    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd" event={"ID":"d1500251-e98d-4183-8b38-dede0b91244e","Type":"ContainerStarted","Data":"d562006e66f04a8997a398c5579a126dd2d6c05085089c31abd431ab63c6e309"}
Oct 30 22:12:17.350611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.350552    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd" podStartSLOduration=2.350538035 podStartE2EDuration="2.350538035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:17.347909351 +0000 UTC m=+1513.682486145" watchObservedRunningTime="2026-10-30 22:12:17.350538035 +0000 UTC m=+1513.685114811"
Oct 30 22:12:17.436589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.436559    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:17.460192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.460160    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7401983b-b5eb-4ac6-9bd8-c02600f955c0-kube-api-access-2tqjb\" (UniqueName: \"kubernetes.io/projected/7401983b-b5eb-4ac6-9bd8-c02600f955c0-kube-api-access-2tqjb\") pod \"7401983b-b5eb-4ac6-9bd8-c02600f955c0\" (UID: \"7401983b-b5eb-4ac6-9bd8-c02600f955c0\") "
Oct 30 22:12:17.462257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.462226    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7401983b-b5eb-4ac6-9bd8-c02600f955c0-kube-api-access-2tqjb" pod "7401983b-b5eb-4ac6-9bd8-c02600f955c0" (UID: "7401983b-b5eb-4ac6-9bd8-c02600f955c0"). InnerVolumeSpecName "kube-api-access-2tqjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:17.562165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.562128    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tqjb\" (UniqueName: \"kubernetes.io/projected/7401983b-b5eb-4ac6-9bd8-c02600f955c0-kube-api-access-2tqjb\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.790543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.790465    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7401983b-b5eb-4ac6-9bd8-c02600f955c0" path="/var/lib/kubelet/pods/7401983b-b5eb-4ac6-9bd8-c02600f955c0/volumes"
Oct 30 22:12:17.858144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:17.858113    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.032646 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.032600    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.187677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.187621    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:18.236982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.236953    5503 generic.go:356] "Generic (PLEG): container finished" podID="d1500251-e98d-4183-8b38-dede0b91244e" containerID="dac4a2349e0abdced608e59259b4c9208257807d780fdb40e57fc7fec8342427" exitCode=0
Oct 30 22:12:18.237427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.237036    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd" event={"ID":"d1500251-e98d-4183-8b38-dede0b91244e","Type":"ContainerDied","Data":"dac4a2349e0abdced608e59259b4c9208257807d780fdb40e57fc7fec8342427"}
Oct 30 22:12:18.238280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.238260    5503 generic.go:356] "Generic (PLEG): container finished" podID="7401983b-b5eb-4ac6-9bd8-c02600f955c0" containerID="49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f" exitCode=137
Oct 30 22:12:18.238358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.238297    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7700-43fbd6"
Oct 30 22:12:18.238358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.238351    5503 scope.go:122] "RemoveContainer" containerID="49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f"
Oct 30 22:12:18.247541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.247518    5503 scope.go:122] "RemoveContainer" containerID="49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f"
Oct 30 22:12:18.247859 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:12:18.247838    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f\": container with ID starting with 49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f not found: ID does not exist" containerID="49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f"
Oct 30 22:12:18.247924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:18.247869    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f"} err="failed to get container status \"49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f\": rpc error: code = NotFound desc = could not find container \"49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f\": container with ID starting with 49c9ad9813a0a6fd695c7b0fd72dac39969e36afe5913e7728b127e86f60840f not found: ID does not exist"
Oct 30 22:12:19.422468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.422444    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:19.492452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.492426    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-openshift-ca\") pod \"d1500251-e98d-4183-8b38-dede0b91244e\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") "
Oct 30 22:12:19.492582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.492539    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1500251-e98d-4183-8b38-dede0b91244e-kube-api-access-wvhld\" (UniqueName: \"kubernetes.io/projected/d1500251-e98d-4183-8b38-dede0b91244e-kube-api-access-wvhld\") pod \"d1500251-e98d-4183-8b38-dede0b91244e\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") "
Oct 30 22:12:19.492719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.492700    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-xks-ca\") pod \"d1500251-e98d-4183-8b38-dede0b91244e\" (UID: \"d1500251-e98d-4183-8b38-dede0b91244e\") "
Oct 30 22:12:19.492871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.492851    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-openshift-ca" pod "d1500251-e98d-4183-8b38-dede0b91244e" (UID: "d1500251-e98d-4183-8b38-dede0b91244e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:19.493158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.493140    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:19.494708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.494683    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1500251-e98d-4183-8b38-dede0b91244e-kube-api-access-wvhld" pod "d1500251-e98d-4183-8b38-dede0b91244e" (UID: "d1500251-e98d-4183-8b38-dede0b91244e"). InnerVolumeSpecName "kube-api-access-wvhld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:19.494796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.494714    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-xks-ca" pod "d1500251-e98d-4183-8b38-dede0b91244e" (UID: "d1500251-e98d-4183-8b38-dede0b91244e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:19.594573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.594497    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhld\" (UniqueName: \"kubernetes.io/projected/d1500251-e98d-4183-8b38-dede0b91244e-kube-api-access-wvhld\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:19.594573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:19.594571    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1500251-e98d-4183-8b38-dede0b91244e-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:20.253544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:20.253509    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"
Oct 30 22:12:20.253742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:20.253548    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd" event={"ID":"d1500251-e98d-4183-8b38-dede0b91244e","Type":"ContainerDied","Data":"d562006e66f04a8997a398c5579a126dd2d6c05085089c31abd431ab63c6e309"}
Oct 30 22:12:20.253742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:20.253602    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d562006e66f04a8997a398c5579a126dd2d6c05085089c31abd431ab63c6e309"
Oct 30 22:12:21.657369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:21.657323    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:24.008334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:24.008302    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:25.051342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:25.051304    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"]
Oct 30 22:12:25.101077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:25.101046    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07dd7c9a-fswfc"]
Oct 30 22:12:25.789311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:25.789278    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38cc1457-a4fe-40fe-8ff6-40321e78ed5e" path="/var/lib/kubelet/pods/38cc1457-a4fe-40fe-8ff6-40321e78ed5e/volumes"
Oct 30 22:12:26.456793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:26.456751    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.008863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:28.008754    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.164026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:28.163971    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:28.264323 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:12:28.264232    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:12:30.480246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:30.480206    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:31.966804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:31.966769    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:32.269535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:32.269431    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:32.431966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:32.431926    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:32.515924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:32.515891    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"]
Oct 30 22:12:32.557835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:32.557750    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d5e5df-zl556"]
Oct 30 22:12:32.972815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:32.972781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:33.790993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:33.790953    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39cbb7a3-59bb-4473-895e-71a1e3f11665" path="/var/lib/kubelet/pods/39cbb7a3-59bb-4473-895e-71a1e3f11665/volumes"
Oct 30 22:12:34.741933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:34.741897    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:35.133487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:35.133400    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:35.500068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:35.500034    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.059406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:36.059368    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"]
Oct 30 22:12:36.112303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:36.112266    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a493d19-pvp8m"]
Oct 30 22:12:36.536186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:36.536150    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:36.750510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:36.747694    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:37.229723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:37.229690    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:37.790570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:37.790528    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd8ad12-362c-49f6-8d64-6fda51e12f86" path="/var/lib/kubelet/pods/fcd8ad12-362c-49f6-8d64-6fda51e12f86/volumes"
Oct 30 22:12:39.872081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:39.872045    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:40.038350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:40.038273    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:40.238523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:40.237927    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:40.409570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:40.409529    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:40.435661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:40.433768    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:12:40.596402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:40.596311    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:41.197662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.196826    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"]
Oct 30 22:12:41.198194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.197678    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d1500251-e98d-4183-8b38-dede0b91244e" containerName="revoke-keys"
Oct 30 22:12:41.198194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.197698    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1500251-e98d-4183-8b38-dede0b91244e" containerName="revoke-keys"
Oct 30 22:12:41.198194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.197859    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1500251-e98d-4183-8b38-dede0b91244e" containerName="revoke-keys"
Oct 30 22:12:41.213325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.211752    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.296677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.268305    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:41.296677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.273183    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"]
Oct 30 22:12:41.321025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.320986    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:12:41.321340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.321319    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:41.321528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.321513    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:41.377662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.373963    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.403597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.403551    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6c5\" (UniqueName: \"kubernetes.io/projected/abfb7457-8705-42f0-8e30-507cb72c8fda-kube-api-access-4j6c5\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.403940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.403915    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.505844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.505748    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6c5\" (UniqueName: \"kubernetes.io/projected/abfb7457-8705-42f0-8e30-507cb72c8fda-kube-api-access-4j6c5\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.505844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.505822    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.506080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.505950    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.506510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.506490    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.508229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.508211    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.594660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.594533    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6c5\" (UniqueName: \"kubernetes.io/projected/abfb7457-8705-42f0-8e30-507cb72c8fda-kube-api-access-4j6c5\") pod \"maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:41.597150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:41.595845    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:42.189696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:42.172932    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"]
Oct 30 22:12:42.500964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:42.500878    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh" event={"ID":"abfb7457-8705-42f0-8e30-507cb72c8fda","Type":"ContainerStarted","Data":"7b19278e28667efe99f4b2f9c587f6246ce30bbfa00faf9c3f3cb9959b293009"}
Oct 30 22:12:42.692764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:42.692648    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:43.508937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:43.508908    5503 generic.go:356] "Generic (PLEG): container finished" podID="abfb7457-8705-42f0-8e30-507cb72c8fda" containerID="1f6c8b99ee0d7737587b56535b7847e7590d059830e27de99610e2e73d50b338" exitCode=0
Oct 30 22:12:43.509310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:43.508988    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh" event={"ID":"abfb7457-8705-42f0-8e30-507cb72c8fda","Type":"ContainerDied","Data":"1f6c8b99ee0d7737587b56535b7847e7590d059830e27de99610e2e73d50b338"}
Oct 30 22:12:43.840211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:43.840108    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:44.056659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.056050    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:44.779340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.779313    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:44.835425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.835393    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abfb7457-8705-42f0-8e30-507cb72c8fda-kube-api-access-4j6c5\" (UniqueName: \"kubernetes.io/projected/abfb7457-8705-42f0-8e30-507cb72c8fda-kube-api-access-4j6c5\") pod \"abfb7457-8705-42f0-8e30-507cb72c8fda\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") "
Oct 30 22:12:44.835626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.835467    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-xks-ca\") pod \"abfb7457-8705-42f0-8e30-507cb72c8fda\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") "
Oct 30 22:12:44.835626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.835575    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-openshift-ca\") pod \"abfb7457-8705-42f0-8e30-507cb72c8fda\" (UID: \"abfb7457-8705-42f0-8e30-507cb72c8fda\") "
Oct 30 22:12:44.835967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.835931    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-openshift-ca" pod "abfb7457-8705-42f0-8e30-507cb72c8fda" (UID: "abfb7457-8705-42f0-8e30-507cb72c8fda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:44.836148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.836129    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:44.837746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.837723    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-xks-ca" pod "abfb7457-8705-42f0-8e30-507cb72c8fda" (UID: "abfb7457-8705-42f0-8e30-507cb72c8fda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:44.837833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.837745    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfb7457-8705-42f0-8e30-507cb72c8fda-kube-api-access-4j6c5" pod "abfb7457-8705-42f0-8e30-507cb72c8fda" (UID: "abfb7457-8705-42f0-8e30-507cb72c8fda"). InnerVolumeSpecName "kube-api-access-4j6c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:44.937882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.937842    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j6c5\" (UniqueName: \"kubernetes.io/projected/abfb7457-8705-42f0-8e30-507cb72c8fda-kube-api-access-4j6c5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:44.937882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:44.937883    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abfb7457-8705-42f0-8e30-507cb72c8fda-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:45.620853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:45.620809    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh" event={"ID":"abfb7457-8705-42f0-8e30-507cb72c8fda","Type":"ContainerDied","Data":"7b19278e28667efe99f4b2f9c587f6246ce30bbfa00faf9c3f3cb9959b293009"}
Oct 30 22:12:45.620853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:45.620850    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"
Oct 30 22:12:45.621064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:45.620899    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b19278e28667efe99f4b2f9c587f6246ce30bbfa00faf9c3f3cb9959b293009"
Oct 30 22:12:47.471204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.471167    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"]
Oct 30 22:12:47.471703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.471689    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abfb7457-8705-42f0-8e30-507cb72c8fda" containerName="revoke-keys"
Oct 30 22:12:47.471756 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.471704    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abfb7457-8705-42f0-8e30-507cb72c8fda" containerName="revoke-keys"
Oct 30 22:12:47.471849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.471840    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abfb7457-8705-42f0-8e30-507cb72c8fda" containerName="revoke-keys"
Oct 30 22:12:47.475328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.475306    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.529785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.529754    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"]
Oct 30 22:12:47.577720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.577690    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:47.577913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.577738    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:47.577913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.577784    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:12:47.579289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.579263    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.579480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.579455    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.579593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.579511    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/8113f99c-1f5b-4006-bb0b-893756ac01f4-kube-api-access-cs7n7\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.579739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.579724    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:47.680495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.680458    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.680710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.680507    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/8113f99c-1f5b-4006-bb0b-893756ac01f4-kube-api-access-cs7n7\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.680710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.680594    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.681322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.681292    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.682839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.682822    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.743118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.743046    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/8113f99c-1f5b-4006-bb0b-893756ac01f4-kube-api-access-cs7n7\") pod \"maas-api-revoke-sub-322f22e6-6w47j\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:47.788989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:47.788959    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:48.000383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:48.000298    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:48.178307 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:12:48.178280    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8113f99c_1f5b_4006_bb0b_893756ac01f4.slice/crio-a7e337d5fc9802afd570d4981ae1c87767bb5495f63c85e0adcff9dbcfbdc544 WatchSource:0}: Error finding container a7e337d5fc9802afd570d4981ae1c87767bb5495f63c85e0adcff9dbcfbdc544: Status 404 returned error can't find the container with id a7e337d5fc9802afd570d4981ae1c87767bb5495f63c85e0adcff9dbcfbdc544
Oct 30 22:12:48.182195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:48.182167    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"]
Oct 30 22:12:48.641604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:48.641567    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j" event={"ID":"8113f99c-1f5b-4006-bb0b-893756ac01f4","Type":"ContainerStarted","Data":"4232531bd6fb498814fc067387033d00257fc2b81d94cf0e49d474ff1a2d158b"}
Oct 30 22:12:48.642003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:48.641683    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j" event={"ID":"8113f99c-1f5b-4006-bb0b-893756ac01f4","Type":"ContainerStarted","Data":"a7e337d5fc9802afd570d4981ae1c87767bb5495f63c85e0adcff9dbcfbdc544"}
Oct 30 22:12:48.762622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:48.762521    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j" podStartSLOduration=1.762504135 podStartE2EDuration="1.762504135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:48.759279488 +0000 UTC m=+1545.093856258" watchObservedRunningTime="2026-10-30 22:12:48.762504135 +0000 UTC m=+1545.097080907"
Oct 30 22:12:49.649064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:49.649033    5503 generic.go:356] "Generic (PLEG): container finished" podID="8113f99c-1f5b-4006-bb0b-893756ac01f4" containerID="4232531bd6fb498814fc067387033d00257fc2b81d94cf0e49d474ff1a2d158b" exitCode=0
Oct 30 22:12:49.649499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:49.649122    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j" event={"ID":"8113f99c-1f5b-4006-bb0b-893756ac01f4","Type":"ContainerDied","Data":"4232531bd6fb498814fc067387033d00257fc2b81d94cf0e49d474ff1a2d158b"}
Oct 30 22:12:50.821047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.821018    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:50.926228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.926136    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-xks-ca\") pod \"8113f99c-1f5b-4006-bb0b-893756ac01f4\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") "
Oct 30 22:12:50.926228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.926172    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-openshift-ca\") pod \"8113f99c-1f5b-4006-bb0b-893756ac01f4\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") "
Oct 30 22:12:50.926458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.926273    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8113f99c-1f5b-4006-bb0b-893756ac01f4-kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/8113f99c-1f5b-4006-bb0b-893756ac01f4-kube-api-access-cs7n7\") pod \"8113f99c-1f5b-4006-bb0b-893756ac01f4\" (UID: \"8113f99c-1f5b-4006-bb0b-893756ac01f4\") "
Oct 30 22:12:50.926590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.926560    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-openshift-ca" pod "8113f99c-1f5b-4006-bb0b-893756ac01f4" (UID: "8113f99c-1f5b-4006-bb0b-893756ac01f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:50.926749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.926730    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:50.928360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.928338    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8113f99c-1f5b-4006-bb0b-893756ac01f4-kube-api-access-cs7n7" pod "8113f99c-1f5b-4006-bb0b-893756ac01f4" (UID: "8113f99c-1f5b-4006-bb0b-893756ac01f4"). InnerVolumeSpecName "kube-api-access-cs7n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:50.928611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:50.928586    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-xks-ca" pod "8113f99c-1f5b-4006-bb0b-893756ac01f4" (UID: "8113f99c-1f5b-4006-bb0b-893756ac01f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:51.028526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:51.028484    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7n7\" (UniqueName: \"kubernetes.io/projected/8113f99c-1f5b-4006-bb0b-893756ac01f4-kube-api-access-cs7n7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:51.028526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:51.028521    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8113f99c-1f5b-4006-bb0b-893756ac01f4-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:51.664193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:51.664153    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"
Oct 30 22:12:51.664363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:51.664191    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j" event={"ID":"8113f99c-1f5b-4006-bb0b-893756ac01f4","Type":"ContainerDied","Data":"a7e337d5fc9802afd570d4981ae1c87767bb5495f63c85e0adcff9dbcfbdc544"}
Oct 30 22:12:51.664363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:51.664277    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7e337d5fc9802afd570d4981ae1c87767bb5495f63c85e0adcff9dbcfbdc544"
Oct 30 22:12:52.885848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:52.885814    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:53.038607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:53.038573    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:53.193700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:53.193487    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:54.054314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:54.054280    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"]
Oct 30 22:12:54.104308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:54.104275    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ba405f9-4b9hn"]
Oct 30 22:12:55.614940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:55.614902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:55.780351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:55.780315    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:55.791032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:55.790994    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047e60c4-077a-4450-b26b-f239cbf5d900" path="/var/lib/kubelet/pods/047e60c4-077a-4450-b26b-f239cbf5d900/volumes"
Oct 30 22:12:57.978699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:12:57.978628    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:12:58.134576 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:12:58.134543    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:13:02.658936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:02.658902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:03.052583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:03.052501    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"]
Oct 30 22:13:03.103429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:03.103395    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dbd9815-8tdrt"]
Oct 30 22:13:03.791380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:03.791349    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0b508c-336f-486a-aad9-5b530c889cff" path="/var/lib/kubelet/pods/7e0b508c-336f-486a-aad9-5b530c889cff/volumes"
Oct 30 22:13:04.173282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:04.173250    5503 scope.go:122] "RemoveContainer" containerID="b925c310d8025298bbce31b97b05a51d4e0781be40f22f54d5d4142cdd53605b"
Oct 30 22:13:04.182880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:04.182860    5503 scope.go:122] "RemoveContainer" containerID="85377f54d361500924c818fa3313e06db60cd0951fe7aea54518743c946e5376"
Oct 30 22:13:07.223894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:07.223832    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:08.251763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:08.251726    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:08.956194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:08.956065    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:09.198168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:09.198122    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:10.151524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:10.151480    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:13:11.929227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:11.929181    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.634990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:12.634947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.822956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:12.822900    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:12.978466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:12.978424    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:16.507711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.507673    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"]
Oct 30 22:13:16.509384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.509355    5503 cpu_manager.go:414] "containerMap: removing container" podUID="8113f99c-1f5b-4006-bb0b-893756ac01f4" containerName="revoke-keys"
Oct 30 22:13:16.509547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.509535    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8113f99c-1f5b-4006-bb0b-893756ac01f4" containerName="revoke-keys"
Oct 30 22:13:16.509954 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.509935    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8113f99c-1f5b-4006-bb0b-893756ac01f4" containerName="revoke-keys"
Oct 30 22:13:16.520587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.520563    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.564367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.564337    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"]
Oct 30 22:13:16.611996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.611969    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:16.612175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.612007    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:16.612175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.612019    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:16.627912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.627889    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:13:16.693469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.693429    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxrm\" (UniqueName: \"kubernetes.io/projected/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-kube-api-access-vwxrm\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.693683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.693555    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.693853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.693834    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.795078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.794974    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.795078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.795038    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxrm\" (UniqueName: \"kubernetes.io/projected/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-kube-api-access-vwxrm\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.795311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.795100    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.795769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.795749    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.797394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.797373    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:16.857184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:16.857151    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxrm\" (UniqueName: \"kubernetes.io/projected/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-kube-api-access-vwxrm\") pod \"maas-api-revoke-sub-107aa622-lhqjq\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:17.134790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:17.134709    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:17.258530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:17.258490    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:17.512707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:17.512669    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"]
Oct 30 22:13:18.215645 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:18.215606    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:18.409240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:18.409205    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:18.593838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:18.593729    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:18.857790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:18.857696    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"]
Oct 30 22:13:18.942251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:18.942213    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"]
Oct 30 22:13:18.942414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:18.942347    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.005325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.005280    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdrt\" (UniqueName: \"kubernetes.io/projected/8413d711-80da-4e96-bc78-43e9e15e1f19-kube-api-access-kzdrt\") pod \"payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.005481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.005387    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413d711-80da-4e96-bc78-43e9e15e1f19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.047327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.047300    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04d404\""
Oct 30 22:13:19.047451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.047401    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04d404-dockercfg-4zqrt\""
Oct 30 22:13:19.106593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.106562    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdrt\" (UniqueName: \"kubernetes.io/projected/8413d711-80da-4e96-bc78-43e9e15e1f19-kube-api-access-kzdrt\") pod \"payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.106764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.106625    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413d711-80da-4e96-bc78-43e9e15e1f19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.107468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.107447    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413d711-80da-4e96-bc78-43e9e15e1f19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.115289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.115211    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq" event={"ID":"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86","Type":"ContainerStarted","Data":"58a281415e2ab71cd87d9f9dabe4d6890aebc1caaa5fd155d774bf2cec77eb9d"}
Oct 30 22:13:19.115289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.115276    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq" event={"ID":"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86","Type":"ContainerStarted","Data":"637c6d03e935e3ef6a6917cb8caf98c55f5ea96968a317935829a83c76a9e4a3"}
Oct 30 22:13:19.169382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.169351    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdrt\" (UniqueName: \"kubernetes.io/projected/8413d711-80da-4e96-bc78-43e9e15e1f19-kube-api-access-kzdrt\") pod \"payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.237249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.237182    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq" podStartSLOduration=3.237151049 podStartE2EDuration="3.237151049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:19.233010192 +0000 UTC m=+1575.567586962" watchObservedRunningTime="2026-10-30 22:13:19.237151049 +0000 UTC m=+1575.571727819"
Oct 30 22:13:19.281990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.281958    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:19.488473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.488441    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:19.659884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:19.659860    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"]
Oct 30 22:13:19.686622 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:13:19.686594    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8413d711_80da_4e96_bc78_43e9e15e1f19.slice/crio-7d2d858a0b2ba07a5230f542150ce0b0ad0de39833c1db6ac7f85ab49f6687e1 WatchSource:0}: Error finding container 7d2d858a0b2ba07a5230f542150ce0b0ad0de39833c1db6ac7f85ab49f6687e1: Status 404 returned error can't find the container with id 7d2d858a0b2ba07a5230f542150ce0b0ad0de39833c1db6ac7f85ab49f6687e1
Oct 30 22:13:20.124972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:20.124885    5503 generic.go:356] "Generic (PLEG): container finished" podID="5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" containerID="58a281415e2ab71cd87d9f9dabe4d6890aebc1caaa5fd155d774bf2cec77eb9d" exitCode=0
Oct 30 22:13:20.124972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:20.124964    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq" event={"ID":"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86","Type":"ContainerDied","Data":"58a281415e2ab71cd87d9f9dabe4d6890aebc1caaa5fd155d774bf2cec77eb9d"}
Oct 30 22:13:20.126509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:20.126481    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w" event={"ID":"8413d711-80da-4e96-bc78-43e9e15e1f19","Type":"ContainerStarted","Data":"8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021"}
Oct 30 22:13:20.126623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:20.126538    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:20.126623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:20.126561    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w" event={"ID":"8413d711-80da-4e96-bc78-43e9e15e1f19","Type":"ContainerStarted","Data":"7d2d858a0b2ba07a5230f542150ce0b0ad0de39833c1db6ac7f85ab49f6687e1"}
Oct 30 22:13:20.366651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:20.366599    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w" podStartSLOduration=2.366585286 podStartE2EDuration="2.366585286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:20.365928097 +0000 UTC m=+1576.700504878" watchObservedRunningTime="2026-10-30 22:13:20.366585286 +0000 UTC m=+1576.701162059"
Oct 30 22:13:21.290192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.290160    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:21.332369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.332335    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-xks-ca\") pod \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") "
Oct 30 22:13:21.332521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.332379    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-kube-api-access-vwxrm\" (UniqueName: \"kubernetes.io/projected/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-kube-api-access-vwxrm\") pod \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") "
Oct 30 22:13:21.332521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.332499    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-openshift-ca\") pod \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\" (UID: \"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86\") "
Oct 30 22:13:21.332858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.332835    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-openshift-ca" pod "5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" (UID: "5062c257-b1ac-42b1-8a9f-9d3fd69bfd86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:21.332972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.332956    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:21.334550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.334522    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-xks-ca" pod "5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" (UID: "5062c257-b1ac-42b1-8a9f-9d3fd69bfd86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:21.334630 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.334576    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-kube-api-access-vwxrm" pod "5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" (UID: "5062c257-b1ac-42b1-8a9f-9d3fd69bfd86"). InnerVolumeSpecName "kube-api-access-vwxrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:21.434252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.434213    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:21.434252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:21.434251    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxrm\" (UniqueName: \"kubernetes.io/projected/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86-kube-api-access-vwxrm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:22.144907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:22.144824    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"
Oct 30 22:13:22.144907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:22.144848    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq" event={"ID":"5062c257-b1ac-42b1-8a9f-9d3fd69bfd86","Type":"ContainerDied","Data":"637c6d03e935e3ef6a6917cb8caf98c55f5ea96968a317935829a83c76a9e4a3"}
Oct 30 22:13:22.145087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:22.144908    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="637c6d03e935e3ef6a6917cb8caf98c55f5ea96968a317935829a83c76a9e4a3"
Oct 30 22:13:22.257013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:22.256980    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:27.127070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:27.127010    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:28.099720 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:13:28.099662    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:13:31.137040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:31.137009    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:13:31.676905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:31.676867    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:31.832685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:31.832647    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:32.004735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:32.004624    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:36.370309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:36.370273    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:39.030307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:39.030270    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:13:41.054849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:41.054810    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:45.713713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:45.713667    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:50.302784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:50.302747    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:50.557492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:50.557405    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:52.394958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.394925    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"]
Oct 30 22:13:52.395495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.395480    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" containerName="revoke-keys"
Oct 30 22:13:52.395543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.395495    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" containerName="revoke-keys"
Oct 30 22:13:52.395606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.395597    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" containerName="revoke-keys"
Oct 30 22:13:52.398912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.398895    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.452854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.452817    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"]
Oct 30 22:13:52.500565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.500532    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:52.500798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.500720    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:13:52.500798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.500784    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:13:52.503538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.503517    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:13:52.505173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.505150    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.505230 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.505185    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.505365 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.505351    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97bs\" (UniqueName: \"kubernetes.io/projected/0c22dbac-7089-4a3b-9c38-74340521c362-kube-api-access-z97bs\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.607175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.607143    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.607378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.607181    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.607378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.607262    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97bs\" (UniqueName: \"kubernetes.io/projected/0c22dbac-7089-4a3b-9c38-74340521c362-kube-api-access-z97bs\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.608012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.607983    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.609728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.609705    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.669538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.669505    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97bs\" (UniqueName: \"kubernetes.io/projected/0c22dbac-7089-4a3b-9c38-74340521c362-kube-api-access-z97bs\") pod \"maas-api-revoke-keys-e2e-api-a-872730-z8q5j\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:52.718991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:52.718956    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:53.096833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:53.096761    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"]
Oct 30 22:13:53.129599 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:13:53.129568    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c22dbac_7089_4a3b_9c38_74340521c362.slice/crio-fdf1c733e74e0fa9ae788865a6909b81566eb36abbd2c48143be87064968da74 WatchSource:0}: Error finding container fdf1c733e74e0fa9ae788865a6909b81566eb36abbd2c48143be87064968da74: Status 404 returned error can't find the container with id fdf1c733e74e0fa9ae788865a6909b81566eb36abbd2c48143be87064968da74
Oct 30 22:13:53.422227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:53.422196    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j" event={"ID":"0c22dbac-7089-4a3b-9c38-74340521c362","Type":"ContainerStarted","Data":"fdf1c733e74e0fa9ae788865a6909b81566eb36abbd2c48143be87064968da74"}
Oct 30 22:13:54.430884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:54.430850    5503 generic.go:356] "Generic (PLEG): container finished" podID="0c22dbac-7089-4a3b-9c38-74340521c362" containerID="8a5dd2bda89dcb6058f6913eaf9fa003f2ab11a43707ccd5122e99a3164e6dc8" exitCode=0
Oct 30 22:13:54.431327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:54.430890    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j" event={"ID":"0c22dbac-7089-4a3b-9c38-74340521c362","Type":"ContainerDied","Data":"8a5dd2bda89dcb6058f6913eaf9fa003f2ab11a43707ccd5122e99a3164e6dc8"}
Oct 30 22:13:55.391233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.391198    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:13:55.708473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.708446    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:55.851976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.851938    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-xks-ca\") pod \"0c22dbac-7089-4a3b-9c38-74340521c362\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") "
Oct 30 22:13:55.852151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.852104    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c22dbac-7089-4a3b-9c38-74340521c362-kube-api-access-z97bs\" (UniqueName: \"kubernetes.io/projected/0c22dbac-7089-4a3b-9c38-74340521c362-kube-api-access-z97bs\") pod \"0c22dbac-7089-4a3b-9c38-74340521c362\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") "
Oct 30 22:13:55.852222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.852149    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-openshift-ca\") pod \"0c22dbac-7089-4a3b-9c38-74340521c362\" (UID: \"0c22dbac-7089-4a3b-9c38-74340521c362\") "
Oct 30 22:13:55.852683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.852653    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-openshift-ca" pod "0c22dbac-7089-4a3b-9c38-74340521c362" (UID: "0c22dbac-7089-4a3b-9c38-74340521c362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:55.854327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.854298    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-xks-ca" pod "0c22dbac-7089-4a3b-9c38-74340521c362" (UID: "0c22dbac-7089-4a3b-9c38-74340521c362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:55.854432 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.854384    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c22dbac-7089-4a3b-9c38-74340521c362-kube-api-access-z97bs" pod "0c22dbac-7089-4a3b-9c38-74340521c362" (UID: "0c22dbac-7089-4a3b-9c38-74340521c362"). InnerVolumeSpecName "kube-api-access-z97bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:55.954084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.954009    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:55.954084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.954036    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97bs\" (UniqueName: \"kubernetes.io/projected/0c22dbac-7089-4a3b-9c38-74340521c362-kube-api-access-z97bs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:55.954084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:55.954047    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c22dbac-7089-4a3b-9c38-74340521c362-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:56.446401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:56.446373    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"
Oct 30 22:13:56.446583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:56.446368    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j" event={"ID":"0c22dbac-7089-4a3b-9c38-74340521c362","Type":"ContainerDied","Data":"fdf1c733e74e0fa9ae788865a6909b81566eb36abbd2c48143be87064968da74"}
Oct 30 22:13:56.446583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:13:56.446505    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf1c733e74e0fa9ae788865a6909b81566eb36abbd2c48143be87064968da74"
Oct 30 22:13:58.098996 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:13:58.098958    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:14:00.028115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:00.028078    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:00.227796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:00.227765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:04.216613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:04.216577    5503 scope.go:122] "RemoveContainer" containerID="fee6c0bd1a321c60a402eecfaaeeff0a6146366f44c1535f587dbac03093ab3a"
Oct 30 22:14:04.227383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:04.227353    5503 scope.go:122] "RemoveContainer" containerID="2ee73be4f0c23101914ef2d466ea6a44d9f3dadb6fc129263e2c8c98888d42a8"
Oct 30 22:14:04.244100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:04.244076    5503 scope.go:122] "RemoveContainer" containerID="ec7123824665d03596bd4226aa7dcbc4b7ac1a892f84e68ead1d44e17da1bddc"
Oct 30 22:14:04.253134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:04.253110    5503 scope.go:122] "RemoveContainer" containerID="06ce2d17b83a9a90898c730f78be4edac1342cb1c8c598b8dec5e7c9acfb3d6b"
Oct 30 22:14:04.396969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:04.396925    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:04.560944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:04.560859    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:05.058876 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:05.058838    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:09.052604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:09.052562    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:10.494254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:10.494217    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:10.726116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:10.726076    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:10.890837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:10.890762    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:10.976167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:10.976128    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"]
Oct 30 22:14:11.026674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:11.026622    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-pd7qd"]
Oct 30 22:14:11.792344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:11.792311    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1500251-e98d-4183-8b38-dede0b91244e" path="/var/lib/kubelet/pods/d1500251-e98d-4183-8b38-dede0b91244e/volumes"
Oct 30 22:14:13.680605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:13.680534    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"]
Oct 30 22:14:13.681063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:13.680777    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" containerID="cri-o://37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4" gracePeriod=30
Oct 30 22:14:15.210563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:15.210517    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:15.562953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:15.562863    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:18.290470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:18.290435    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:18.290910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:18.290508    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:18.746546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:18.746521    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:18.944725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:18.944667    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:14:19.030826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.030792    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-socket\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.030835    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-certs\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.030878    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-kube-api-access-p2pbq\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-kube-api-access-p2pbq\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.030911    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-data\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.030929    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-podinfo\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.030959    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-envoy\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031075    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-token\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-token\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031132    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-credential-socket\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031150    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-socket" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:19.031303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031182    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istiod-ca-cert\") pod \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\" (UID: \"d24d013e-a975-46ed-9e2e-9109a7bfaea1\") "
Oct 30 22:14:19.031303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031200    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-data" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:19.031303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031300    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-certs" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:19.031594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031377    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-credential-socket" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:19.031594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031564    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istiod-ca-cert" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:19.031841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031812    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.031841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031838    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.032025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031852    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.032025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031867    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.032025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.031883    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.033345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.033317    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-podinfo" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:19.033582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.033539    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-token" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:19.033688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.033614    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-kube-api-access-p2pbq" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "kube-api-access-p2pbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:19.033739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.033680    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-envoy" pod "d24d013e-a975-46ed-9e2e-9109a7bfaea1" (UID: "d24d013e-a975-46ed-9e2e-9109a7bfaea1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:19.133915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.133872    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pbq\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-kube-api-access-p2pbq\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.133915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.133915    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.134106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.133931    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.134106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.133946    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d013e-a975-46ed-9e2e-9109a7bfaea1-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:19.706858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.706827    5503 generic.go:356] "Generic (PLEG): container finished" podID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerID="37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4" exitCode=0
Oct 30 22:14:19.707309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.706878    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"
Oct 30 22:14:19.707309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.706911    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" event={"ID":"d24d013e-a975-46ed-9e2e-9109a7bfaea1","Type":"ContainerDied","Data":"37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4"}
Oct 30 22:14:19.707309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.706966    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph" event={"ID":"d24d013e-a975-46ed-9e2e-9109a7bfaea1","Type":"ContainerDied","Data":"b3034a277c823c6dd2b8fdb537f4023f08c8f49da6c4217a04455ab9fd0e0a12"}
Oct 30 22:14:19.707309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.706993    5503 scope.go:122] "RemoveContainer" containerID="37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4"
Oct 30 22:14:19.728191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.728169    5503 scope.go:122] "RemoveContainer" containerID="37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4"
Oct 30 22:14:19.728506 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:14:19.728484    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4\": container with ID starting with 37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4 not found: ID does not exist" containerID="37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4"
Oct 30 22:14:19.728580 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.728518    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4"} err="failed to get container status \"37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4\": rpc error: code = NotFound desc = could not find container \"37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4\": container with ID starting with 37bfa2f25594219b916ef39d0610b35ae42f45257bc2f9299eec6dfdf3a2eba4 not found: ID does not exist"
Oct 30 22:14:19.822716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.822686    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:19.928715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.928675    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"]
Oct 30 22:14:19.979173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:19.979098    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-x5qph"]
Oct 30 22:14:20.525550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:20.525502    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:21.789183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:21.789149    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" path="/var/lib/kubelet/pods/d24d013e-a975-46ed-9e2e-9109a7bfaea1/volumes"
Oct 30 22:14:22.365516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:22.365480    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:14:22.999307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:22.999268    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:23.159197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:23.159165    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:25.545668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:25.545036    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"]
Oct 30 22:14:25.545668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:25.545287    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy" containerID="cri-o://5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46" gracePeriod=30
Oct 30 22:14:25.553209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:25.553164    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:25.829089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:25.828937    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:27.587867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:27.587831    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:28.099467 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:14:28.099427    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:14:29.066918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.066885    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"]
Oct 30 22:14:29.067415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.067402    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy"
Oct 30 22:14:29.067459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.067418    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy"
Oct 30 22:14:29.067459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.067428    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0c22dbac-7089-4a3b-9c38-74340521c362" containerName="revoke-keys"
Oct 30 22:14:29.067459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.067432    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c22dbac-7089-4a3b-9c38-74340521c362" containerName="revoke-keys"
Oct 30 22:14:29.067551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.067539    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24d013e-a975-46ed-9e2e-9109a7bfaea1" containerName="istio-proxy"
Oct 30 22:14:29.067551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.067546    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c22dbac-7089-4a3b-9c38-74340521c362" containerName="revoke-keys"
Oct 30 22:14:29.071119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.071102    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.125593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.125563    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"]
Oct 30 22:14:29.148370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.148338    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.148528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.148476    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.148570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.148558    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwg9j\" (UniqueName: \"kubernetes.io/projected/3304d763-6c48-431f-b065-e66788f30a4c-kube-api-access-fwg9j\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.173325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.173297    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:29.173479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.173332    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:14:29.173479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.173345    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:14:29.175206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.175190    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:14:29.250810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.250781    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwg9j\" (UniqueName: \"kubernetes.io/projected/3304d763-6c48-431f-b065-e66788f30a4c-kube-api-access-fwg9j\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.250993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.250871    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.250993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.250948    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.251587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.251568    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.253271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.253250    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.313655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.313614    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwg9j\" (UniqueName: \"kubernetes.io/projected/3304d763-6c48-431f-b065-e66788f30a4c-kube-api-access-fwg9j\") pod \"maas-api-revoke-keys-e2e-api-a-161111-6z4rc\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.384199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.384127    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:29.755001 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:14:29.754971    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3304d763_6c48_431f_b065_e66788f30a4c.slice/crio-c72947a7ecbdfd1e5621eca1cd1a6906c077f2c04672a50b6c33ae7d8a551ee1 WatchSource:0}: Error finding container c72947a7ecbdfd1e5621eca1cd1a6906c077f2c04672a50b6c33ae7d8a551ee1: Status 404 returned error can't find the container with id c72947a7ecbdfd1e5621eca1cd1a6906c077f2c04672a50b6c33ae7d8a551ee1
Oct 30 22:14:29.766618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.760570    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"]
Oct 30 22:14:29.912454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:29.912419    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc" event={"ID":"3304d763-6c48-431f-b065-e66788f30a4c","Type":"ContainerStarted","Data":"c72947a7ecbdfd1e5621eca1cd1a6906c077f2c04672a50b6c33ae7d8a551ee1"}
Oct 30 22:14:30.269445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.269415    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:30.831371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.831315    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:14:30.920467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.920435    5503 generic.go:356] "Generic (PLEG): container finished" podID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerID="5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46" exitCode=0
Oct 30 22:14:30.920659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.920485    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"
Oct 30 22:14:30.920659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.920513    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" event={"ID":"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a","Type":"ContainerDied","Data":"5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46"}
Oct 30 22:14:30.920659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.920560    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8" event={"ID":"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a","Type":"ContainerDied","Data":"ec866f2dfaf2f946e61a32e856af00ccd0c7e487d5f98d68beb363b1ef20e167"}
Oct 30 22:14:30.920659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.920584    5503 scope.go:122] "RemoveContainer" containerID="5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46"
Oct 30 22:14:30.922129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.922106    5503 generic.go:356] "Generic (PLEG): container finished" podID="3304d763-6c48-431f-b065-e66788f30a4c" containerID="95352813a2922d1a0ac62304ebb285ad8195ecd554892692d26dd8ebf19d572a" exitCode=0
Oct 30 22:14:30.922249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.922189    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc" event={"ID":"3304d763-6c48-431f-b065-e66788f30a4c","Type":"ContainerDied","Data":"95352813a2922d1a0ac62304ebb285ad8195ecd554892692d26dd8ebf19d572a"}
Oct 30 22:14:30.930788 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.930767    5503 scope.go:122] "RemoveContainer" containerID="5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46"
Oct 30 22:14:30.931091 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:14:30.931072    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46\": container with ID starting with 5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46 not found: ID does not exist" containerID="5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46"
Oct 30 22:14:30.931137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.931098    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46"} err="failed to get container status \"5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46\": rpc error: code = NotFound desc = could not find container \"5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46\": container with ID starting with 5e19ddfbda738dd58ef4ed25603c4a4b4d39db073712f2a679dab9c8105c2d46 not found: ID does not exist"
Oct 30 22:14:30.975493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975457    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-envoy\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.975610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975512    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-podinfo\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.975610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975569    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istiod-ca-cert\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.975610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975602    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-data\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.975763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975709    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-credential-socket\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.975919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975881    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-kube-api-access-tnrpp\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-kube-api-access-tnrpp\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.976010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975920    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-token\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-token\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.976010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.975927    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-data" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.976119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976012    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-certs\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.976119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976040    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-credential-socket" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.976119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976051    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istiod-ca-cert" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:30.976119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976088    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-socket\") pod \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\" (UID: \"b08c39ab-2daf-4f0f-a2f6-51824ad30b1a\") "
Oct 30 22:14:30.976316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976286    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-certs" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.976394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976376    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-socket" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.976760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976737    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:30.976819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976768    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:30.976819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976783    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:30.976819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976799    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:30.976819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.976814    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:30.977753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.977733    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-envoy" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:30.977977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.977957    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-podinfo" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:30.978369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.978354    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-token" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:30.978410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:30.978366    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-kube-api-access-tnrpp" pod "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" (UID: "b08c39ab-2daf-4f0f-a2f6-51824ad30b1a"). InnerVolumeSpecName "kube-api-access-tnrpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:31.078201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:31.078166    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.078201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:31.078198    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.078406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:31.078209    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrpp\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-kube-api-access-tnrpp\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.078406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:31.078219    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:31.345988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:31.345955    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"]
Oct 30 22:14:31.350134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:31.350110    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-dtsz8"]
Oct 30 22:14:31.790208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:31.790177    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" path="/var/lib/kubelet/pods/b08c39ab-2daf-4f0f-a2f6-51824ad30b1a/volumes"
Oct 30 22:14:32.067332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.067306    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:32.191384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.191343    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-openshift-ca\") pod \"3304d763-6c48-431f-b065-e66788f30a4c\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") "
Oct 30 22:14:32.191588 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.191563    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3304d763-6c48-431f-b065-e66788f30a4c-kube-api-access-fwg9j\" (UniqueName: \"kubernetes.io/projected/3304d763-6c48-431f-b065-e66788f30a4c-kube-api-access-fwg9j\") pod \"3304d763-6c48-431f-b065-e66788f30a4c\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") "
Oct 30 22:14:32.191873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.191852    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-xks-ca\") pod \"3304d763-6c48-431f-b065-e66788f30a4c\" (UID: \"3304d763-6c48-431f-b065-e66788f30a4c\") "
Oct 30 22:14:32.192545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.192515    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-openshift-ca" pod "3304d763-6c48-431f-b065-e66788f30a4c" (UID: "3304d763-6c48-431f-b065-e66788f30a4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:32.194201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.194176    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-xks-ca" pod "3304d763-6c48-431f-b065-e66788f30a4c" (UID: "3304d763-6c48-431f-b065-e66788f30a4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:32.194337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.194319    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3304d763-6c48-431f-b065-e66788f30a4c-kube-api-access-fwg9j" pod "3304d763-6c48-431f-b065-e66788f30a4c" (UID: "3304d763-6c48-431f-b065-e66788f30a4c"). InnerVolumeSpecName "kube-api-access-fwg9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:32.293299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.293262    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwg9j\" (UniqueName: \"kubernetes.io/projected/3304d763-6c48-431f-b065-e66788f30a4c-kube-api-access-fwg9j\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:32.293299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.293301    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:32.293514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.293313    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3304d763-6c48-431f-b065-e66788f30a4c-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:32.939049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.939019    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"
Oct 30 22:14:32.939049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.939041    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc" event={"ID":"3304d763-6c48-431f-b065-e66788f30a4c","Type":"ContainerDied","Data":"c72947a7ecbdfd1e5621eca1cd1a6906c077f2c04672a50b6c33ae7d8a551ee1"}
Oct 30 22:14:32.939514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:32.939088    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72947a7ecbdfd1e5621eca1cd1a6906c077f2c04672a50b6c33ae7d8a551ee1"
Oct 30 22:14:34.299927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:34.299898    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:35.050919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:35.050888    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"]
Oct 30 22:14:35.132020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:35.131958    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adb60c4-75pj7"]
Oct 30 22:14:35.790402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:35.790373    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104a5f93-9413-4823-b60b-7e9dab5232e2" path="/var/lib/kubelet/pods/104a5f93-9413-4823-b60b-7e9dab5232e2/volumes"
Oct 30 22:14:35.919062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:35.919026    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:36.126459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:36.126370    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:36.391489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:36.391413    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:36.591065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:36.591025    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"]
Oct 30 22:14:36.641550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:36.641476    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dbe0a7-s7czh"]
Oct 30 22:14:37.035495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:37.035466    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:37.789083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:37.789053    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abfb7457-8705-42f0-8e30-507cb72c8fda" path="/var/lib/kubelet/pods/abfb7457-8705-42f0-8e30-507cb72c8fda/volumes"
Oct 30 22:14:40.768808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:40.768775    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:41.004382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:41.004347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:41.156942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:41.156911    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:41.314429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:41.314399    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:43.423755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:43.423717    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:43.591688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:43.591630    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:43.754472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:43.754390    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:44.773784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:44.773751    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:45.432369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:45.432315    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:45.666407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:45.666369    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:45.828969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:45.828891    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:50.039794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:50.039756    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:50.244597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:50.244561    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:50.418410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:50.418374    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:50.640832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:50.640800    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:53.463698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:53.463671    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:53.678659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:53.658057    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:55.795379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:55.795330    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:14:58.099438 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:14:58.099403    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:14:59.830582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:14:59.830544    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"]
Oct 30 22:15:00.086726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.086614    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.191016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.190980    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"]
Oct 30 22:15:00.192138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.192109    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy"
Oct 30 22:15:00.192311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.192293    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy"
Oct 30 22:15:00.192417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.192404    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3304d763-6c48-431f-b065-e66788f30a4c" containerName="revoke-keys"
Oct 30 22:15:00.192513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.192501    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3304d763-6c48-431f-b065-e66788f30a4c" containerName="revoke-keys"
Oct 30 22:15:00.192772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.192753    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08c39ab-2daf-4f0f-a2f6-51824ad30b1a" containerName="istio-proxy"
Oct 30 22:15:00.192903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.192890    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3304d763-6c48-431f-b065-e66788f30a4c" containerName="revoke-keys"
Oct 30 22:15:00.197704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.197679    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.200154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.200127    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"]
Oct 30 22:15:00.216077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.216048    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"]
Oct 30 22:15:00.216893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.216869    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.220851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.220826    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.238317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.238297    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"]
Oct 30 22:15:00.242848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.242828    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.246586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.246566    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"]
Oct 30 22:15:00.250232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.250214    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"]
Oct 30 22:15:00.250360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.250340    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.254024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.254001    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.255104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.255084    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:15:00.255191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.255140    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:15:00.258147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.258124    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:15:00.291384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.291356    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"]
Oct 30 22:15:00.293045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.293020    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.294195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.294177    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"]
Oct 30 22:15:00.297110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.297090    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"]
Oct 30 22:15:00.302362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.302339    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4l2f\" (UniqueName: \"kubernetes.io/projected/b9758362-f06d-44af-8e1c-6013d5d3ab13-kube-api-access-b4l2f\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.302473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.302401    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5973f004-7df1-498f-b740-29a227a3860c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.302473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.302433    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9758362-f06d-44af-8e1c-6013d5d3ab13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.302601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.302583    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpntw\" (UniqueName: \"kubernetes.io/projected/9213ee9a-5bfb-49c1-bee6-385569d0ab70-kube-api-access-rpntw\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.302776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.302758    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9213ee9a-5bfb-49c1-bee6-385569d0ab70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.302822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.302801    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcj95\" (UniqueName: \"kubernetes.io/projected/5973f004-7df1-498f-b740-29a227a3860c-kube-api-access-fcj95\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.320951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.320918    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"]
Oct 30 22:15:00.337811 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.331071    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"]
Oct 30 22:15:00.354013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.348949    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"]
Oct 30 22:15:00.404262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404104    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9213ee9a-5bfb-49c1-bee6-385569d0ab70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.404262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404181    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcj95\" (UniqueName: \"kubernetes.io/projected/5973f004-7df1-498f-b740-29a227a3860c-kube-api-access-fcj95\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404284    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6ql\" (UniqueName: \"kubernetes.io/projected/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-kube-api-access-vg6ql\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404334    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4l2f\" (UniqueName: \"kubernetes.io/projected/b9758362-f06d-44af-8e1c-6013d5d3ab13-kube-api-access-b4l2f\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404377    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-kube-api-access-cwz5v\") pod \"maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404419    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5973f004-7df1-498f-b740-29a227a3860c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404445    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9758362-f06d-44af-8e1c-6013d5d3ab13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404517    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxvx\" (UniqueName: \"kubernetes.io/projected/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-kube-api-access-bvxvx\") pod \"maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404542    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404590    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpntw\" (UniqueName: \"kubernetes.io/projected/9213ee9a-5bfb-49c1-bee6-385569d0ab70-kube-api-access-rpntw\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.404748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404674    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.405374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.404832    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.405445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.405424    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9758362-f06d-44af-8e1c-6013d5d3ab13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.406059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.406038    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5973f004-7df1-498f-b740-29a227a3860c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.406131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.406109    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9213ee9a-5bfb-49c1-bee6-385569d0ab70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.472004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.471970    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4l2f\" (UniqueName: \"kubernetes.io/projected/b9758362-f06d-44af-8e1c-6013d5d3ab13-kube-api-access-b4l2f\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.473019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.472990    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcj95\" (UniqueName: \"kubernetes.io/projected/5973f004-7df1-498f-b740-29a227a3860c-kube-api-access-fcj95\") pod \"maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.481927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.481905    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:00.485710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.485690    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpntw\" (UniqueName: \"kubernetes.io/projected/9213ee9a-5bfb-49c1-bee6-385569d0ab70-kube-api-access-rpntw\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.522714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.515673    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.523282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.523245    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6ql\" (UniqueName: \"kubernetes.io/projected/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-kube-api-access-vg6ql\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.526814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.526783    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-kube-api-access-cwz5v\") pod \"maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.527107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.527085    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxvx\" (UniqueName: \"kubernetes.io/projected/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-kube-api-access-bvxvx\") pod \"maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.527683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.527664    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.527877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.527847    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.529042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.528952    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.534070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.529375    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:00.534289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.516459    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.534407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.533227    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.538343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.538321    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:00.547797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.547768    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:00.595433 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.595365    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6ql\" (UniqueName: \"kubernetes.io/projected/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-kube-api-access-vg6ql\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.595433 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.595379    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxvx\" (UniqueName: \"kubernetes.io/projected/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-kube-api-access-bvxvx\") pod \"maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.596021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.595998    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-kube-api-access-cwz5v\") pod \"maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.856379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.856306    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:00.866782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.866757    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:00.875310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.875283    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:00.905664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:00.905546    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:01.035967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.035876    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"]
Oct 30 22:15:01.044219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.043082    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"]
Oct 30 22:15:01.062672 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:15:01.062074    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5973f004_7df1_498f_b740_29a227a3860c.slice/crio-03a4ceb5465bd7030762a888715c58a62fe6513db93a4c3b1de7b746ec754db6 WatchSource:0}: Error finding container 03a4ceb5465bd7030762a888715c58a62fe6513db93a4c3b1de7b746ec754db6: Status 404 returned error can't find the container with id 03a4ceb5465bd7030762a888715c58a62fe6513db93a4c3b1de7b746ec754db6
Oct 30 22:15:01.083082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.083059    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:01.147267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.132187    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"]
Oct 30 22:15:01.243509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.243470    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" event={"ID":"b9758362-f06d-44af-8e1c-6013d5d3ab13","Type":"ContainerStarted","Data":"967b48d050d3a993db746bf6b027d1b5c4d92d3f80f65ee0a10fca74ea134394"}
Oct 30 22:15:01.243902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.243845    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:01.245097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.245070    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" event={"ID":"9213ee9a-5bfb-49c1-bee6-385569d0ab70","Type":"ContainerStarted","Data":"b90bd1dd81ec7be7c473f0abd4313c21afc0c1fb5158f4e0a5ec029b9f3ce536"}
Oct 30 22:15:01.246536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.246509    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" event={"ID":"5973f004-7df1-498f-b740-29a227a3860c","Type":"ContainerStarted","Data":"03a4ceb5465bd7030762a888715c58a62fe6513db93a4c3b1de7b746ec754db6"}
Oct 30 22:15:01.248992 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:15:01.248960    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ae1a9c_59b0_415f_8c62_89fd4701e9f4.slice/crio-2eaf2281991258444c004b88701b91339da00acae3c771afe9445bf4daf0fede WatchSource:0}: Error finding container 2eaf2281991258444c004b88701b91339da00acae3c771afe9445bf4daf0fede: Status 404 returned error can't find the container with id 2eaf2281991258444c004b88701b91339da00acae3c771afe9445bf4daf0fede
Oct 30 22:15:01.251585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.251535    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"]
Oct 30 22:15:01.453816 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:15:01.453785    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7feb5e7_40f9_4a32_b6dc_b11e1d2b694a.slice/crio-8aadb62b8653fe5e6ea594a07d910511198a9a8c96be1d949cf59bf7da7ddb70 WatchSource:0}: Error finding container 8aadb62b8653fe5e6ea594a07d910511198a9a8c96be1d949cf59bf7da7ddb70: Status 404 returned error can't find the container with id 8aadb62b8653fe5e6ea594a07d910511198a9a8c96be1d949cf59bf7da7ddb70
Oct 30 22:15:01.457511 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.457487    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"]
Oct 30 22:15:01.577717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:01.570707    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"]
Oct 30 22:15:02.255804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.255778    5503 generic.go:356] "Generic (PLEG): container finished" podID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerID="dd86b3d70779bad5f146c6834f81924d1baf0ac1b3fcbde2ca3b5b762213a2f0" exitCode=60
Oct 30 22:15:02.256186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.255830    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" event={"ID":"9213ee9a-5bfb-49c1-bee6-385569d0ab70","Type":"ContainerDied","Data":"dd86b3d70779bad5f146c6834f81924d1baf0ac1b3fcbde2ca3b5b762213a2f0"}
Oct 30 22:15:02.256186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.256152    5503 scope.go:122] "RemoveContainer" containerID="dd86b3d70779bad5f146c6834f81924d1baf0ac1b3fcbde2ca3b5b762213a2f0"
Oct 30 22:15:02.257518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.257499    5503 generic.go:356] "Generic (PLEG): container finished" podID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerID="2bb6a5a9c0adb2d52bff3a8a0d7b7906cd78bf97b6c322d0d61a3a4e491cc360" exitCode=60
Oct 30 22:15:02.257651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.257576    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" event={"ID":"06ae1a9c-59b0-415f-8c62-89fd4701e9f4","Type":"ContainerDied","Data":"2bb6a5a9c0adb2d52bff3a8a0d7b7906cd78bf97b6c322d0d61a3a4e491cc360"}
Oct 30 22:15:02.257651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.257621    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" event={"ID":"06ae1a9c-59b0-415f-8c62-89fd4701e9f4","Type":"ContainerStarted","Data":"2eaf2281991258444c004b88701b91339da00acae3c771afe9445bf4daf0fede"}
Oct 30 22:15:02.257872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.257853    5503 scope.go:122] "RemoveContainer" containerID="2bb6a5a9c0adb2d52bff3a8a0d7b7906cd78bf97b6c322d0d61a3a4e491cc360"
Oct 30 22:15:02.259263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.259240    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerStarted","Data":"0028b3926ed99b3106be56451f024a9bde61be04fbe9e84d22211511893cfbe3"}
Oct 30 22:15:02.259350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.259287    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerStarted","Data":"20d0532b51bb32928632e757e7a1bc69d510e2055e872c3c590de8142f16afd3"}
Oct 30 22:15:02.260974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.260958    5503 generic.go:356] "Generic (PLEG): container finished" podID="5973f004-7df1-498f-b740-29a227a3860c" containerID="999ae218baf1b18736863626ce7299bc9fb878bfcc6760ddb1606a31012b7ae4" exitCode=60
Oct 30 22:15:02.261113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.261035    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" event={"ID":"5973f004-7df1-498f-b740-29a227a3860c","Type":"ContainerDied","Data":"999ae218baf1b18736863626ce7299bc9fb878bfcc6760ddb1606a31012b7ae4"}
Oct 30 22:15:02.261274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.261258    5503 scope.go:122] "RemoveContainer" containerID="999ae218baf1b18736863626ce7299bc9fb878bfcc6760ddb1606a31012b7ae4"
Oct 30 22:15:02.262694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.262674    5503 generic.go:356] "Generic (PLEG): container finished" podID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerID="9ddc8179deb951ac6aee2bdac2fe626d89222c5adef495f149f6edfbadeb6f5e" exitCode=60
Oct 30 22:15:02.262784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.262748    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" event={"ID":"b9758362-f06d-44af-8e1c-6013d5d3ab13","Type":"ContainerDied","Data":"9ddc8179deb951ac6aee2bdac2fe626d89222c5adef495f149f6edfbadeb6f5e"}
Oct 30 22:15:02.262972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.262956    5503 scope.go:122] "RemoveContainer" containerID="9ddc8179deb951ac6aee2bdac2fe626d89222c5adef495f149f6edfbadeb6f5e"
Oct 30 22:15:02.264303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.264284    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerStarted","Data":"8850d56d33e8bf275c14cde7be9fed093b5c29b1347e88d32b5b418c10ac2ab6"}
Oct 30 22:15:02.264385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.264329    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerStarted","Data":"8aadb62b8653fe5e6ea594a07d910511198a9a8c96be1d949cf59bf7da7ddb70"}
Oct 30 22:15:02.886205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:02.886131    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" podStartSLOduration=2.886106509 podStartE2EDuration="2.886106509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.878016823 +0000 UTC m=+1679.212593593" watchObservedRunningTime="2026-10-30 22:15:02.886106509 +0000 UTC m=+1679.220683282"
Oct 30 22:15:03.010557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.010488    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" podStartSLOduration=3.010465154 podStartE2EDuration="3.010465154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.008266438 +0000 UTC m=+1679.342843220" watchObservedRunningTime="2026-10-30 22:15:03.010465154 +0000 UTC m=+1679.345041925"
Oct 30 22:15:03.274246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.274210    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" event={"ID":"b9758362-f06d-44af-8e1c-6013d5d3ab13","Type":"ContainerStarted","Data":"bc066831c80862ddb1081318fc29dd19f1d3d883869e567cd8f4fd6a6fb80bcb"}
Oct 30 22:15:03.276003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.275977    5503 generic.go:356] "Generic (PLEG): container finished" podID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerID="8850d56d33e8bf275c14cde7be9fed093b5c29b1347e88d32b5b418c10ac2ab6" exitCode=60
Oct 30 22:15:03.276135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.276023    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerDied","Data":"8850d56d33e8bf275c14cde7be9fed093b5c29b1347e88d32b5b418c10ac2ab6"}
Oct 30 22:15:03.276303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.276289    5503 scope.go:122] "RemoveContainer" containerID="8850d56d33e8bf275c14cde7be9fed093b5c29b1347e88d32b5b418c10ac2ab6"
Oct 30 22:15:03.278101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.278069    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" event={"ID":"9213ee9a-5bfb-49c1-bee6-385569d0ab70","Type":"ContainerStarted","Data":"b8b5ff0d7566b91748fa2bd6b6f2fe7d53a3751a7c6fb9d2ff90945c957a8bf8"}
Oct 30 22:15:03.280110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.280081    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" event={"ID":"06ae1a9c-59b0-415f-8c62-89fd4701e9f4","Type":"ContainerStarted","Data":"c9ec6eca0cb8bb229f09b4f0af8cf661de1b6a5636f78bedb6387690d675568d"}
Oct 30 22:15:03.281749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.281727    5503 generic.go:356] "Generic (PLEG): container finished" podID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerID="0028b3926ed99b3106be56451f024a9bde61be04fbe9e84d22211511893cfbe3" exitCode=60
Oct 30 22:15:03.281839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.281792    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerDied","Data":"0028b3926ed99b3106be56451f024a9bde61be04fbe9e84d22211511893cfbe3"}
Oct 30 22:15:03.281984 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.281968    5503 scope.go:122] "RemoveContainer" containerID="0028b3926ed99b3106be56451f024a9bde61be04fbe9e84d22211511893cfbe3"
Oct 30 22:15:03.284803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.284779    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" event={"ID":"5973f004-7df1-498f-b740-29a227a3860c","Type":"ContainerStarted","Data":"779056878b9caeed8924b7c2891aa17eba081b32dc9ff9aa6fbac6547f8fe7e7"}
Oct 30 22:15:03.402013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.401962    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" podStartSLOduration=3.401946832 podStartE2EDuration="3.401946832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.401072253 +0000 UTC m=+1679.735649022" watchObservedRunningTime="2026-10-30 22:15:03.401946832 +0000 UTC m=+1679.736523601"
Oct 30 22:15:03.661328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.661271    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" podStartSLOduration=3.661257848 podStartE2EDuration="3.661257848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.65978436 +0000 UTC m=+1679.994361128" watchObservedRunningTime="2026-10-30 22:15:03.661257848 +0000 UTC m=+1679.995834616"
Oct 30 22:15:03.662882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.662845    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" podStartSLOduration=3.6628354400000003 podStartE2EDuration="3.66283544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.534610807 +0000 UTC m=+1679.869187575" watchObservedRunningTime="2026-10-30 22:15:03.66283544 +0000 UTC m=+1679.997412209"
Oct 30 22:15:03.802689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:03.789256    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" podStartSLOduration=3.7892381630000003 podStartE2EDuration="3.789238163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.788772229 +0000 UTC m=+1680.123348999" watchObservedRunningTime="2026-10-30 22:15:03.789238163 +0000 UTC m=+1680.123814935"
Oct 30 22:15:04.293759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.293672    5503 generic.go:356] "Generic (PLEG): container finished" podID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerID="bc066831c80862ddb1081318fc29dd19f1d3d883869e567cd8f4fd6a6fb80bcb" exitCode=60
Oct 30 22:15:04.294197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.293759    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" event={"ID":"b9758362-f06d-44af-8e1c-6013d5d3ab13","Type":"ContainerDied","Data":"bc066831c80862ddb1081318fc29dd19f1d3d883869e567cd8f4fd6a6fb80bcb"}
Oct 30 22:15:04.294197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.293859    5503 scope.go:122] "RemoveContainer" containerID="9ddc8179deb951ac6aee2bdac2fe626d89222c5adef495f149f6edfbadeb6f5e"
Oct 30 22:15:04.294197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.294044    5503 scope.go:122] "RemoveContainer" containerID="bc066831c80862ddb1081318fc29dd19f1d3d883869e567cd8f4fd6a6fb80bcb"
Oct 30 22:15:04.294406 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:04.294383    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs_odh-ai-gateway-infra(b9758362-f06d-44af-8e1c-6013d5d3ab13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13"
Oct 30 22:15:04.295894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.295871    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerStarted","Data":"5aab80de7caf9152e20187722ebf29e96f1ce3f983e06ca06b9126bf9da96180"}
Oct 30 22:15:04.298007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.297980    5503 generic.go:356] "Generic (PLEG): container finished" podID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerID="b8b5ff0d7566b91748fa2bd6b6f2fe7d53a3751a7c6fb9d2ff90945c957a8bf8" exitCode=60
Oct 30 22:15:04.298117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.298057    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" event={"ID":"9213ee9a-5bfb-49c1-bee6-385569d0ab70","Type":"ContainerDied","Data":"b8b5ff0d7566b91748fa2bd6b6f2fe7d53a3751a7c6fb9d2ff90945c957a8bf8"}
Oct 30 22:15:04.298318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.298298    5503 scope.go:122] "RemoveContainer" containerID="b8b5ff0d7566b91748fa2bd6b6f2fe7d53a3751a7c6fb9d2ff90945c957a8bf8"
Oct 30 22:15:04.301012 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:04.298574    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r_odh-ai-gateway-infra(9213ee9a-5bfb-49c1-bee6-385569d0ab70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70"
Oct 30 22:15:04.301012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.299966    5503 generic.go:356] "Generic (PLEG): container finished" podID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerID="c9ec6eca0cb8bb229f09b4f0af8cf661de1b6a5636f78bedb6387690d675568d" exitCode=60
Oct 30 22:15:04.301012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.299988    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" event={"ID":"06ae1a9c-59b0-415f-8c62-89fd4701e9f4","Type":"ContainerDied","Data":"c9ec6eca0cb8bb229f09b4f0af8cf661de1b6a5636f78bedb6387690d675568d"}
Oct 30 22:15:04.301012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.300296    5503 scope.go:122] "RemoveContainer" containerID="c9ec6eca0cb8bb229f09b4f0af8cf661de1b6a5636f78bedb6387690d675568d"
Oct 30 22:15:04.301012 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:04.300573    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt_odh-ai-gateway-infra(06ae1a9c-59b0-415f-8c62-89fd4701e9f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4"
Oct 30 22:15:04.302126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.302102    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerStarted","Data":"7b2577b41f6284e23f62fa7b6960f7fae010870dc08ebe8d35739b046fdc77af"}
Oct 30 22:15:04.303931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.303907    5503 generic.go:356] "Generic (PLEG): container finished" podID="5973f004-7df1-498f-b740-29a227a3860c" containerID="779056878b9caeed8924b7c2891aa17eba081b32dc9ff9aa6fbac6547f8fe7e7" exitCode=60
Oct 30 22:15:04.304022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.303976    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" event={"ID":"5973f004-7df1-498f-b740-29a227a3860c","Type":"ContainerDied","Data":"779056878b9caeed8924b7c2891aa17eba081b32dc9ff9aa6fbac6547f8fe7e7"}
Oct 30 22:15:04.304183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.304166    5503 scope.go:122] "RemoveContainer" containerID="779056878b9caeed8924b7c2891aa17eba081b32dc9ff9aa6fbac6547f8fe7e7"
Oct 30 22:15:04.304401 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:04.304382    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6_odh-ai-gateway-infra(5973f004-7df1-498f-b740-29a227a3860c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" podUID="5973f004-7df1-498f-b740-29a227a3860c"
Oct 30 22:15:04.312708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.312687    5503 scope.go:122] "RemoveContainer" containerID="dd86b3d70779bad5f146c6834f81924d1baf0ac1b3fcbde2ca3b5b762213a2f0"
Oct 30 22:15:04.314567 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:04.314544    5503 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r_odh-ai-gateway-infra_cleanup-dd86b3d70779bad5f146c6834f81924d1baf0ac1b3fcbde2ca3b5b762213a2f0.log: no such file or directory" path="/var/log/containers/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r_odh-ai-gateway-infra_cleanup-dd86b3d70779bad5f146c6834f81924d1baf0ac1b3fcbde2ca3b5b762213a2f0.log"
Oct 30 22:15:04.325355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.325336    5503 scope.go:122] "RemoveContainer" containerID="2bb6a5a9c0adb2d52bff3a8a0d7b7906cd78bf97b6c322d0d61a3a4e491cc360"
Oct 30 22:15:04.335765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.335748    5503 scope.go:122] "RemoveContainer" containerID="999ae218baf1b18736863626ce7299bc9fb878bfcc6760ddb1606a31012b7ae4"
Oct 30 22:15:04.830940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:04.830887    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w" podUID="8413d711-80da-4e96-bc78-43e9e15e1f19" containerName="payload-processing" containerID="cri-o://8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021" gracePeriod=25
Oct 30 22:15:05.097852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.097798    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:15:05.263115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.263085    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8413d711-80da-4e96-bc78-43e9e15e1f19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413d711-80da-4e96-bc78-43e9e15e1f19-plugins-config-volume\") pod \"8413d711-80da-4e96-bc78-43e9e15e1f19\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") "
Oct 30 22:15:05.263280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.263148    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8413d711-80da-4e96-bc78-43e9e15e1f19-kube-api-access-kzdrt\" (UniqueName: \"kubernetes.io/projected/8413d711-80da-4e96-bc78-43e9e15e1f19-kube-api-access-kzdrt\") pod \"8413d711-80da-4e96-bc78-43e9e15e1f19\" (UID: \"8413d711-80da-4e96-bc78-43e9e15e1f19\") "
Oct 30 22:15:05.263545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.263514    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8413d711-80da-4e96-bc78-43e9e15e1f19-plugins-config-volume" pod "8413d711-80da-4e96-bc78-43e9e15e1f19" (UID: "8413d711-80da-4e96-bc78-43e9e15e1f19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:05.265426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.265396    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8413d711-80da-4e96-bc78-43e9e15e1f19-kube-api-access-kzdrt" pod "8413d711-80da-4e96-bc78-43e9e15e1f19" (UID: "8413d711-80da-4e96-bc78-43e9e15e1f19"). InnerVolumeSpecName "kube-api-access-kzdrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:05.312912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.312883    5503 generic.go:356] "Generic (PLEG): container finished" podID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerID="7b2577b41f6284e23f62fa7b6960f7fae010870dc08ebe8d35739b046fdc77af" exitCode=60
Oct 30 22:15:05.313263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.312971    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerDied","Data":"7b2577b41f6284e23f62fa7b6960f7fae010870dc08ebe8d35739b046fdc77af"}
Oct 30 22:15:05.313263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.313031    5503 scope.go:122] "RemoveContainer" containerID="0028b3926ed99b3106be56451f024a9bde61be04fbe9e84d22211511893cfbe3"
Oct 30 22:15:05.313263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.313237    5503 scope.go:122] "RemoveContainer" containerID="7b2577b41f6284e23f62fa7b6960f7fae010870dc08ebe8d35739b046fdc77af"
Oct 30 22:15:05.313565 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:05.313545    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q_odh-ai-gateway-infra(cb2cd334-3fae-467a-ad8f-cc3b15c26b73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73"
Oct 30 22:15:05.314996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.314982    5503 scope.go:122] "RemoveContainer" containerID="779056878b9caeed8924b7c2891aa17eba081b32dc9ff9aa6fbac6547f8fe7e7"
Oct 30 22:15:05.315266 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:05.315243    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6_odh-ai-gateway-infra(5973f004-7df1-498f-b740-29a227a3860c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" podUID="5973f004-7df1-498f-b740-29a227a3860c"
Oct 30 22:15:05.316143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.316117    5503 generic.go:356] "Generic (PLEG): container finished" podID="8413d711-80da-4e96-bc78-43e9e15e1f19" containerID="8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021" exitCode=0
Oct 30 22:15:05.316242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.316149    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"
Oct 30 22:15:05.316242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.316197    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w" event={"ID":"8413d711-80da-4e96-bc78-43e9e15e1f19","Type":"ContainerDied","Data":"8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021"}
Oct 30 22:15:05.316353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.316247    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w" event={"ID":"8413d711-80da-4e96-bc78-43e9e15e1f19","Type":"ContainerDied","Data":"7d2d858a0b2ba07a5230f542150ce0b0ad0de39833c1db6ac7f85ab49f6687e1"}
Oct 30 22:15:05.320188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.320170    5503 scope.go:122] "RemoveContainer" containerID="bc066831c80862ddb1081318fc29dd19f1d3d883869e567cd8f4fd6a6fb80bcb"
Oct 30 22:15:05.320475 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:05.320450    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs_odh-ai-gateway-infra(b9758362-f06d-44af-8e1c-6013d5d3ab13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13"
Oct 30 22:15:05.324253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.324230    5503 generic.go:356] "Generic (PLEG): container finished" podID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerID="5aab80de7caf9152e20187722ebf29e96f1ce3f983e06ca06b9126bf9da96180" exitCode=60
Oct 30 22:15:05.324344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.324316    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerDied","Data":"5aab80de7caf9152e20187722ebf29e96f1ce3f983e06ca06b9126bf9da96180"}
Oct 30 22:15:05.324758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.324556    5503 scope.go:122] "RemoveContainer" containerID="5aab80de7caf9152e20187722ebf29e96f1ce3f983e06ca06b9126bf9da96180"
Oct 30 22:15:05.325166 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:05.325139    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx_odh-ai-gateway-infra(d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a"
Oct 30 22:15:05.326689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.326591    5503 scope.go:122] "RemoveContainer" containerID="8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021"
Oct 30 22:15:05.326848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.326811    5503 scope.go:122] "RemoveContainer" containerID="b8b5ff0d7566b91748fa2bd6b6f2fe7d53a3751a7c6fb9d2ff90945c957a8bf8"
Oct 30 22:15:05.327115 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:05.327089    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r_odh-ai-gateway-infra(9213ee9a-5bfb-49c1-bee6-385569d0ab70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70"
Oct 30 22:15:05.328519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.328497    5503 scope.go:122] "RemoveContainer" containerID="c9ec6eca0cb8bb229f09b4f0af8cf661de1b6a5636f78bedb6387690d675568d"
Oct 30 22:15:05.328805 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:05.328783    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt_odh-ai-gateway-infra(06ae1a9c-59b0-415f-8c62-89fd4701e9f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4"
Oct 30 22:15:05.335628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.335610    5503 scope.go:122] "RemoveContainer" containerID="8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021"
Oct 30 22:15:05.335907 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:05.335888    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021\": container with ID starting with 8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021 not found: ID does not exist" containerID="8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021"
Oct 30 22:15:05.335949 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.335914    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021"} err="failed to get container status \"8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021\": rpc error: code = NotFound desc = could not find container \"8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021\": container with ID starting with 8dcd3e80f107aa78cb7c35104066f9119c185ec69439b7123244760f5af4a021 not found: ID does not exist"
Oct 30 22:15:05.335949 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.335930    5503 scope.go:122] "RemoveContainer" containerID="8850d56d33e8bf275c14cde7be9fed093b5c29b1347e88d32b5b418c10ac2ab6"
Oct 30 22:15:05.364169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.364122    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413d711-80da-4e96-bc78-43e9e15e1f19-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:05.364169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.364143    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdrt\" (UniqueName: \"kubernetes.io/projected/8413d711-80da-4e96-bc78-43e9e15e1f19-kube-api-access-kzdrt\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:05.547630 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.547599    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"]
Oct 30 22:15:05.555827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.555805    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04d404-7c76b54d77-6jq8w"]
Oct 30 22:15:05.790973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.790940    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8413d711-80da-4e96-bc78-43e9e15e1f19" path="/var/lib/kubelet/pods/8413d711-80da-4e96-bc78-43e9e15e1f19/volumes"
Oct 30 22:15:05.847883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:05.847836    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:06.337128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:06.337099    5503 scope.go:122] "RemoveContainer" containerID="5aab80de7caf9152e20187722ebf29e96f1ce3f983e06ca06b9126bf9da96180"
Oct 30 22:15:06.337578 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:06.337386    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx_odh-ai-gateway-infra(d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a"
Oct 30 22:15:06.338732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:06.338701    5503 scope.go:122] "RemoveContainer" containerID="7b2577b41f6284e23f62fa7b6960f7fae010870dc08ebe8d35739b046fdc77af"
Oct 30 22:15:06.338960 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:06.338936    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q_odh-ai-gateway-infra(cb2cd334-3fae-467a-ad8f-cc3b15c26b73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73"
Oct 30 22:15:09.917520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:09.917487    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.365388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:11.365358    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.659860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:11.659828    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.821514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:11.821483    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:11.885654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:11.885612    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"]
Oct 30 22:15:11.936181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:11.936094    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-872730-z8q5j"]
Oct 30 22:15:12.292843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:12.292759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:13.790317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:13.790283    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c22dbac-7089-4a3b-9c38-74340521c362" path="/var/lib/kubelet/pods/0c22dbac-7089-4a3b-9c38-74340521c362/volumes"
Oct 30 22:15:14.781910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:14.781883    5503 scope.go:122] "RemoveContainer" containerID="b8b5ff0d7566b91748fa2bd6b6f2fe7d53a3751a7c6fb9d2ff90945c957a8bf8"
Oct 30 22:15:15.398774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:15.398692    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" event={"ID":"9213ee9a-5bfb-49c1-bee6-385569d0ab70","Type":"ContainerStarted","Data":"f5f72bf38a11dc37b81313984ba3bc5ccc17426b8aaee347f860ca8f4b248ef5"}
Oct 30 22:15:15.785167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:15.785131    5503 scope.go:122] "RemoveContainer" containerID="c9ec6eca0cb8bb229f09b4f0af8cf661de1b6a5636f78bedb6387690d675568d"
Oct 30 22:15:16.150962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.150935    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:15:16.407067 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.407035    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" event={"ID":"06ae1a9c-59b0-415f-8c62-89fd4701e9f4","Type":"ContainerStarted","Data":"05329f9feba90e80b62acfb689aff17467752ecaae2ff37fe583931674deb430"}
Oct 30 22:15:16.408938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.408911    5503 generic.go:356] "Generic (PLEG): container finished" podID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerID="f5f72bf38a11dc37b81313984ba3bc5ccc17426b8aaee347f860ca8f4b248ef5" exitCode=60
Oct 30 22:15:16.409058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.408951    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" event={"ID":"9213ee9a-5bfb-49c1-bee6-385569d0ab70","Type":"ContainerDied","Data":"f5f72bf38a11dc37b81313984ba3bc5ccc17426b8aaee347f860ca8f4b248ef5"}
Oct 30 22:15:16.409058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.408991    5503 scope.go:122] "RemoveContainer" containerID="b8b5ff0d7566b91748fa2bd6b6f2fe7d53a3751a7c6fb9d2ff90945c957a8bf8"
Oct 30 22:15:16.409337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.409319    5503 scope.go:122] "RemoveContainer" containerID="f5f72bf38a11dc37b81313984ba3bc5ccc17426b8aaee347f860ca8f4b248ef5"
Oct 30 22:15:16.409665 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:16.409614    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r_odh-ai-gateway-infra(9213ee9a-5bfb-49c1-bee6-385569d0ab70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70"
Oct 30 22:15:16.545660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.545063    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"]
Oct 30 22:15:16.653462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.653416    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:16.782805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.782723    5503 scope.go:122] "RemoveContainer" containerID="bc066831c80862ddb1081318fc29dd19f1d3d883869e567cd8f4fd6a6fb80bcb"
Oct 30 22:15:16.782975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.782893    5503 scope.go:122] "RemoveContainer" containerID="779056878b9caeed8924b7c2891aa17eba081b32dc9ff9aa6fbac6547f8fe7e7"
Oct 30 22:15:16.783099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:16.783081    5503 scope.go:122] "RemoveContainer" containerID="7b2577b41f6284e23f62fa7b6960f7fae010870dc08ebe8d35739b046fdc77af"
Oct 30 22:15:17.423761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.423718    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" event={"ID":"5973f004-7df1-498f-b740-29a227a3860c","Type":"ContainerStarted","Data":"4867d0970880176078c9082e1b18003d2fc546ba8c039decd722f1d56d823118"}
Oct 30 22:15:17.425677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.425626    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" event={"ID":"b9758362-f06d-44af-8e1c-6013d5d3ab13","Type":"ContainerStarted","Data":"a7bab121308c689bdaa7369ac2901b5e4aaee097e6a4f44f43f5306081ff6eb1"}
Oct 30 22:15:17.428969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.428947    5503 generic.go:356] "Generic (PLEG): container finished" podID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerID="05329f9feba90e80b62acfb689aff17467752ecaae2ff37fe583931674deb430" exitCode=60
Oct 30 22:15:17.429066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.429027    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" event={"ID":"06ae1a9c-59b0-415f-8c62-89fd4701e9f4","Type":"ContainerDied","Data":"05329f9feba90e80b62acfb689aff17467752ecaae2ff37fe583931674deb430"}
Oct 30 22:15:17.429126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.429085    5503 scope.go:122] "RemoveContainer" containerID="c9ec6eca0cb8bb229f09b4f0af8cf661de1b6a5636f78bedb6387690d675568d"
Oct 30 22:15:17.429577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.429537    5503 scope.go:122] "RemoveContainer" containerID="05329f9feba90e80b62acfb689aff17467752ecaae2ff37fe583931674deb430"
Oct 30 22:15:17.429917 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:17.429891    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt_odh-ai-gateway-infra(06ae1a9c-59b0-415f-8c62-89fd4701e9f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4"
Oct 30 22:15:17.431008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.430988    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerStarted","Data":"91ac727bf72656b9086e3bbc56ba8e74355d6666ddee1fee01104dae6a6bb1ab"}
Oct 30 22:15:17.541370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.541338    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"]
Oct 30 22:15:17.578157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.578130    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:17.650079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.650046    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9213ee9a-5bfb-49c1-bee6-385569d0ab70-kube-api-access-rpntw\" (UniqueName: \"kubernetes.io/projected/9213ee9a-5bfb-49c1-bee6-385569d0ab70-kube-api-access-rpntw\") pod \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") "
Oct 30 22:15:17.650279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.650195    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9213ee9a-5bfb-49c1-bee6-385569d0ab70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9213ee9a-5bfb-49c1-bee6-385569d0ab70-maas-api-service-ca\") pod \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\" (UID: \"9213ee9a-5bfb-49c1-bee6-385569d0ab70\") "
Oct 30 22:15:17.650605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.650578    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9213ee9a-5bfb-49c1-bee6-385569d0ab70-maas-api-service-ca" pod "9213ee9a-5bfb-49c1-bee6-385569d0ab70" (UID: "9213ee9a-5bfb-49c1-bee6-385569d0ab70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:17.652262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.652243    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9213ee9a-5bfb-49c1-bee6-385569d0ab70-kube-api-access-rpntw" pod "9213ee9a-5bfb-49c1-bee6-385569d0ab70" (UID: "9213ee9a-5bfb-49c1-bee6-385569d0ab70"). InnerVolumeSpecName "kube-api-access-rpntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:17.751404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.751357    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpntw\" (UniqueName: \"kubernetes.io/projected/9213ee9a-5bfb-49c1-bee6-385569d0ab70-kube-api-access-rpntw\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:17.751404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:17.751388    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9213ee9a-5bfb-49c1-bee6-385569d0ab70-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.439103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.439071    5503 generic.go:356] "Generic (PLEG): container finished" podID="5973f004-7df1-498f-b740-29a227a3860c" containerID="4867d0970880176078c9082e1b18003d2fc546ba8c039decd722f1d56d823118" exitCode=60
Oct 30 22:15:18.439571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.439149    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" event={"ID":"5973f004-7df1-498f-b740-29a227a3860c","Type":"ContainerDied","Data":"4867d0970880176078c9082e1b18003d2fc546ba8c039decd722f1d56d823118"}
Oct 30 22:15:18.439571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.439214    5503 scope.go:122] "RemoveContainer" containerID="779056878b9caeed8924b7c2891aa17eba081b32dc9ff9aa6fbac6547f8fe7e7"
Oct 30 22:15:18.439685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.439651    5503 scope.go:122] "RemoveContainer" containerID="4867d0970880176078c9082e1b18003d2fc546ba8c039decd722f1d56d823118"
Oct 30 22:15:18.440026 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:18.440001    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6_odh-ai-gateway-infra(5973f004-7df1-498f-b740-29a227a3860c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" podUID="5973f004-7df1-498f-b740-29a227a3860c"
Oct 30 22:15:18.441421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.441403    5503 generic.go:356] "Generic (PLEG): container finished" podID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerID="a7bab121308c689bdaa7369ac2901b5e4aaee097e6a4f44f43f5306081ff6eb1" exitCode=60
Oct 30 22:15:18.441538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.441492    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" event={"ID":"b9758362-f06d-44af-8e1c-6013d5d3ab13","Type":"ContainerDied","Data":"a7bab121308c689bdaa7369ac2901b5e4aaee097e6a4f44f43f5306081ff6eb1"}
Oct 30 22:15:18.441870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.441851    5503 scope.go:122] "RemoveContainer" containerID="a7bab121308c689bdaa7369ac2901b5e4aaee097e6a4f44f43f5306081ff6eb1"
Oct 30 22:15:18.442186 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:18.442163    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs_odh-ai-gateway-infra(b9758362-f06d-44af-8e1c-6013d5d3ab13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13"
Oct 30 22:15:18.442791 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.442774    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"
Oct 30 22:15:18.442885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.442771    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r" event={"ID":"9213ee9a-5bfb-49c1-bee6-385569d0ab70","Type":"ContainerDied","Data":"b90bd1dd81ec7be7c473f0abd4313c21afc0c1fb5158f4e0a5ec029b9f3ce536"}
Oct 30 22:15:18.446725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.446704    5503 generic.go:356] "Generic (PLEG): container finished" podID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerID="91ac727bf72656b9086e3bbc56ba8e74355d6666ddee1fee01104dae6a6bb1ab" exitCode=60
Oct 30 22:15:18.446819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.446731    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerDied","Data":"91ac727bf72656b9086e3bbc56ba8e74355d6666ddee1fee01104dae6a6bb1ab"}
Oct 30 22:15:18.447040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.447024    5503 scope.go:122] "RemoveContainer" containerID="91ac727bf72656b9086e3bbc56ba8e74355d6666ddee1fee01104dae6a6bb1ab"
Oct 30 22:15:18.447315 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:18.447278    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q_odh-ai-gateway-infra(cb2cd334-3fae-467a-ad8f-cc3b15c26b73)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73"
Oct 30 22:15:18.454129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.454112    5503 scope.go:122] "RemoveContainer" containerID="bc066831c80862ddb1081318fc29dd19f1d3d883869e567cd8f4fd6a6fb80bcb"
Oct 30 22:15:18.467319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.467298    5503 scope.go:122] "RemoveContainer" containerID="f5f72bf38a11dc37b81313984ba3bc5ccc17426b8aaee347f860ca8f4b248ef5"
Oct 30 22:15:18.476986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.476967    5503 scope.go:122] "RemoveContainer" containerID="7b2577b41f6284e23f62fa7b6960f7fae010870dc08ebe8d35739b046fdc77af"
Oct 30 22:15:18.553557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.553527    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"]
Oct 30 22:15:18.627737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.627715    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:18.661581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.661555    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-maas-api-service-ca\") pod \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") "
Oct 30 22:15:18.661757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.661596    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-kube-api-access-bvxvx\" (UniqueName: \"kubernetes.io/projected/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-kube-api-access-bvxvx\") pod \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\" (UID: \"06ae1a9c-59b0-415f-8c62-89fd4701e9f4\") "
Oct 30 22:15:18.662038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.661963    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-maas-api-service-ca" pod "06ae1a9c-59b0-415f-8c62-89fd4701e9f4" (UID: "06ae1a9c-59b0-415f-8c62-89fd4701e9f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.662205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.662190    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.664279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.663874    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-kube-api-access-bvxvx" pod "06ae1a9c-59b0-415f-8c62-89fd4701e9f4" (UID: "06ae1a9c-59b0-415f-8c62-89fd4701e9f4"). InnerVolumeSpecName "kube-api-access-bvxvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.672004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.671980    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"]
Oct 30 22:15:18.763931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.763858    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxvx\" (UniqueName: \"kubernetes.io/projected/06ae1a9c-59b0-415f-8c62-89fd4701e9f4-kube-api-access-bvxvx\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.781907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.781883    5503 scope.go:122] "RemoveContainer" containerID="5aab80de7caf9152e20187722ebf29e96f1ce3f983e06ca06b9126bf9da96180"
Oct 30 22:15:18.809847 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.809825    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"]
Oct 30 22:15:18.821536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:18.821514    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889975-f6b2r"]
Oct 30 22:15:19.030529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.030494    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"]
Oct 30 22:15:19.095430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.095400    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:19.457361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.457325    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerStarted","Data":"a16feec701ce93a7b21c38670d21651db2d278eb6689a9088e90294c17f0c6f3"}
Oct 30 22:15:19.459687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.459661    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt" event={"ID":"06ae1a9c-59b0-415f-8c62-89fd4701e9f4","Type":"ContainerDied","Data":"2eaf2281991258444c004b88701b91339da00acae3c771afe9445bf4daf0fede"}
Oct 30 22:15:19.459831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.459674    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"
Oct 30 22:15:19.459831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.459711    5503 scope.go:122] "RemoveContainer" containerID="05329f9feba90e80b62acfb689aff17467752ecaae2ff37fe583931674deb430"
Oct 30 22:15:19.689058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.689027    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:19.783875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.783842    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9758362-f06d-44af-8e1c-6013d5d3ab13-kube-api-access-b4l2f\" (UniqueName: \"kubernetes.io/projected/b9758362-f06d-44af-8e1c-6013d5d3ab13-kube-api-access-b4l2f\") pod \"b9758362-f06d-44af-8e1c-6013d5d3ab13\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") "
Oct 30 22:15:19.784038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.783902    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9758362-f06d-44af-8e1c-6013d5d3ab13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9758362-f06d-44af-8e1c-6013d5d3ab13-maas-api-service-ca\") pod \"b9758362-f06d-44af-8e1c-6013d5d3ab13\" (UID: \"b9758362-f06d-44af-8e1c-6013d5d3ab13\") "
Oct 30 22:15:19.784277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.784248    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9758362-f06d-44af-8e1c-6013d5d3ab13-maas-api-service-ca" pod "b9758362-f06d-44af-8e1c-6013d5d3ab13" (UID: "b9758362-f06d-44af-8e1c-6013d5d3ab13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.784489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.784471    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9758362-f06d-44af-8e1c-6013d5d3ab13-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.786044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.786014    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9758362-f06d-44af-8e1c-6013d5d3ab13-kube-api-access-b4l2f" pod "b9758362-f06d-44af-8e1c-6013d5d3ab13" (UID: "b9758362-f06d-44af-8e1c-6013d5d3ab13"). InnerVolumeSpecName "kube-api-access-b4l2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.790442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.790416    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" path="/var/lib/kubelet/pods/9213ee9a-5bfb-49c1-bee6-385569d0ab70/volumes"
Oct 30 22:15:19.816489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.816466    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:19.820843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.820826    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:19.838487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.838458    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"]
Oct 30 22:15:19.842227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.842207    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-161111-29889975-mwbpt"]
Oct 30 22:15:19.884845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.884816    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5973f004-7df1-498f-b740-29a227a3860c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5973f004-7df1-498f-b740-29a227a3860c-maas-api-service-ca\") pod \"5973f004-7df1-498f-b740-29a227a3860c\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") "
Oct 30 22:15:19.884990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.884945    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5973f004-7df1-498f-b740-29a227a3860c-kube-api-access-fcj95\" (UniqueName: \"kubernetes.io/projected/5973f004-7df1-498f-b740-29a227a3860c-kube-api-access-fcj95\") pod \"5973f004-7df1-498f-b740-29a227a3860c\" (UID: \"5973f004-7df1-498f-b740-29a227a3860c\") "
Oct 30 22:15:19.885108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.885065    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-kube-api-access-vg6ql\" (UniqueName: \"kubernetes.io/projected/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-kube-api-access-vg6ql\") pod \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") "
Oct 30 22:15:19.885179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.885134    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5973f004-7df1-498f-b740-29a227a3860c-maas-api-service-ca" pod "5973f004-7df1-498f-b740-29a227a3860c" (UID: "5973f004-7df1-498f-b740-29a227a3860c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.885236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.885195    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-maas-api-service-ca\") pod \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\" (UID: \"cb2cd334-3fae-467a-ad8f-cc3b15c26b73\") "
Oct 30 22:15:19.885522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.885502    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-maas-api-service-ca" pod "cb2cd334-3fae-467a-ad8f-cc3b15c26b73" (UID: "cb2cd334-3fae-467a-ad8f-cc3b15c26b73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.885748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.885731    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5973f004-7df1-498f-b740-29a227a3860c-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.885822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.885755    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4l2f\" (UniqueName: \"kubernetes.io/projected/b9758362-f06d-44af-8e1c-6013d5d3ab13-kube-api-access-b4l2f\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.885822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.885771    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.887152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.887130    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-kube-api-access-vg6ql" pod "cb2cd334-3fae-467a-ad8f-cc3b15c26b73" (UID: "cb2cd334-3fae-467a-ad8f-cc3b15c26b73"). InnerVolumeSpecName "kube-api-access-vg6ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.887199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.887165    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5973f004-7df1-498f-b740-29a227a3860c-kube-api-access-fcj95" pod "5973f004-7df1-498f-b740-29a227a3860c" (UID: "5973f004-7df1-498f-b740-29a227a3860c"). InnerVolumeSpecName "kube-api-access-fcj95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.924833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.924788    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:19.987863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.987222    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6ql\" (UniqueName: \"kubernetes.io/projected/cb2cd334-3fae-467a-ad8f-cc3b15c26b73-kube-api-access-vg6ql\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.987863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:19.987261    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcj95\" (UniqueName: \"kubernetes.io/projected/5973f004-7df1-498f-b740-29a227a3860c-kube-api-access-fcj95\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.111613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.111570    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:20.284829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.284733    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"]
Oct 30 22:15:20.284989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.284938    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy" containerID="cri-o://6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd" gracePeriod=30
Oct 30 22:15:20.288614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.288579    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:20.481701 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.458473    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:20.544198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.544115    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs" event={"ID":"b9758362-f06d-44af-8e1c-6013d5d3ab13","Type":"ContainerDied","Data":"967b48d050d3a993db746bf6b027d1b5c4d92d3f80f65ee0a10fca74ea134394"}
Oct 30 22:15:20.544198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.544194    5503 scope.go:122] "RemoveContainer" containerID="a7bab121308c689bdaa7369ac2901b5e4aaee097e6a4f44f43f5306081ff6eb1"
Oct 30 22:15:20.544375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.544332    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"
Oct 30 22:15:20.549740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.549712    5503 generic.go:356] "Generic (PLEG): container finished" podID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerID="a16feec701ce93a7b21c38670d21651db2d278eb6689a9088e90294c17f0c6f3" exitCode=60
Oct 30 22:15:20.549856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.549814    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerDied","Data":"a16feec701ce93a7b21c38670d21651db2d278eb6689a9088e90294c17f0c6f3"}
Oct 30 22:15:20.550141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.550128    5503 scope.go:122] "RemoveContainer" containerID="a16feec701ce93a7b21c38670d21651db2d278eb6689a9088e90294c17f0c6f3"
Oct 30 22:15:20.550395 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:20.550378    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx_odh-ai-gateway-infra(d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a"
Oct 30 22:15:20.553211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.553180    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"
Oct 30 22:15:20.553315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.553211    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q" event={"ID":"cb2cd334-3fae-467a-ad8f-cc3b15c26b73","Type":"ContainerDied","Data":"20d0532b51bb32928632e757e7a1bc69d510e2055e872c3c590de8142f16afd3"}
Oct 30 22:15:20.554557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.554538    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6" event={"ID":"5973f004-7df1-498f-b740-29a227a3860c","Type":"ContainerDied","Data":"03a4ceb5465bd7030762a888715c58a62fe6513db93a4c3b1de7b746ec754db6"}
Oct 30 22:15:20.554666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.554617    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"
Oct 30 22:15:20.563128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.563104    5503 scope.go:122] "RemoveContainer" containerID="5aab80de7caf9152e20187722ebf29e96f1ce3f983e06ca06b9126bf9da96180"
Oct 30 22:15:20.582523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.582497    5503 scope.go:122] "RemoveContainer" containerID="91ac727bf72656b9086e3bbc56ba8e74355d6666ddee1fee01104dae6a6bb1ab"
Oct 30 22:15:20.591004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.590987    5503 scope.go:122] "RemoveContainer" containerID="4867d0970880176078c9082e1b18003d2fc546ba8c039decd722f1d56d823118"
Oct 30 22:15:20.760850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.760819    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"]
Oct 30 22:15:20.764531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.764504    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889975-92zxs"]
Oct 30 22:15:20.876240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.876170    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"]
Oct 30 22:15:20.880578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.880552    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b47e8d-29889975-qb9w6"]
Oct 30 22:15:20.984573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:20.984544    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"]
Oct 30 22:15:21.123291 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.123261    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"]
Oct 30 22:15:21.152527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.152492    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889975-9wp4q"]
Oct 30 22:15:21.191921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.191891    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"]
Oct 30 22:15:21.192446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192428    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192449    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192464    5503 cpu_manager.go:414] "containerMap: removing container" podUID="8413d711-80da-4e96-bc78-43e9e15e1f19" containerName="payload-processing"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192473    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8413d711-80da-4e96-bc78-43e9e15e1f19" containerName="payload-processing"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192485    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192492    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192505    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192512    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:21.192534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192531    5503 cpu_manager.go:414] "containerMap: removing container" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192538    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192554    5503 cpu_manager.go:414] "containerMap: removing container" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192561    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192574    5503 cpu_manager.go:414] "containerMap: removing container" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192581    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192599    5503 cpu_manager.go:414] "containerMap: removing container" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192606    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192616    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192623    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192675    5503 cpu_manager.go:414] "containerMap: removing container" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192683    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192695    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192702    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192714    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192721    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192732    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192738    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192886    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192898    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8413d711-80da-4e96-bc78-43e9e15e1f19" containerName="payload-processing"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192911    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192921    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192931    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192940    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192949    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192958    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192968    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:21.192970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192980    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:21.194178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192990    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:21.194178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.192999    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:21.198219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.198195    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.251107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.251038    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"]
Oct 30 22:15:21.251816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.251785    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:21.331879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.331849    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.332055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.331924    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22p7\" (UniqueName: \"kubernetes.io/projected/e73ee2dc-660d-4f05-9543-42f0ac7380ed-kube-api-access-z22p7\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.332055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.332007    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.433576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.433492    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.433751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.433593    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22p7\" (UniqueName: \"kubernetes.io/projected/e73ee2dc-660d-4f05-9543-42f0ac7380ed-kube-api-access-z22p7\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.433751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.433698    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.434243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.434216    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.435948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.435932    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.499087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.499053    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22p7\" (UniqueName: \"kubernetes.io/projected/e73ee2dc-660d-4f05-9543-42f0ac7380ed-kube-api-access-z22p7\") pod \"maas-api-revoke-keys-e2e-api-b-4401bb-m299p\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.511880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.511856    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:21.666716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.665420    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:21.790044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.789960    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" path="/var/lib/kubelet/pods/06ae1a9c-59b0-415f-8c62-89fd4701e9f4/volumes"
Oct 30 22:15:21.790510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.790490    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5973f004-7df1-498f-b740-29a227a3860c" path="/var/lib/kubelet/pods/5973f004-7df1-498f-b740-29a227a3860c/volumes"
Oct 30 22:15:21.791286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.791269    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" path="/var/lib/kubelet/pods/b9758362-f06d-44af-8e1c-6013d5d3ab13/volumes"
Oct 30 22:15:21.791930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.791897    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" path="/var/lib/kubelet/pods/cb2cd334-3fae-467a-ad8f-cc3b15c26b73/volumes"
Oct 30 22:15:21.811683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.811663    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:21.895251 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:15:21.895210    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73ee2dc_660d_4f05_9543_42f0ac7380ed.slice/crio-3cb9e164205e4d7dbebb1f3a01e8e05b78159b764acb6965531d900df8201287 WatchSource:0}: Error finding container 3cb9e164205e4d7dbebb1f3a01e8e05b78159b764acb6965531d900df8201287: Status 404 returned error can't find the container with id 3cb9e164205e4d7dbebb1f3a01e8e05b78159b764acb6965531d900df8201287
Oct 30 22:15:21.896720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.896697    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"]
Oct 30 22:15:21.966062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.966021    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-maas-api-service-ca\") pod \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") "
Oct 30 22:15:21.966254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.966231    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-kube-api-access-cwz5v\") pod \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\" (UID: \"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a\") "
Oct 30 22:15:21.966476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.966445    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-maas-api-service-ca" pod "d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" (UID: "d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.966818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.966794    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.968381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:21.968347    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-kube-api-access-cwz5v" pod "d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" (UID: "d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a"). InnerVolumeSpecName "kube-api-access-cwz5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:22.068102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.068019    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a-kube-api-access-cwz5v\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.576698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.576662    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p" event={"ID":"e73ee2dc-660d-4f05-9543-42f0ac7380ed","Type":"ContainerStarted","Data":"2d3c89194cc53faf7aa26df32cd70f3d53de2ff41afd63557758b1b30affb98c"}
Oct 30 22:15:22.577178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.576727    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p" event={"ID":"e73ee2dc-660d-4f05-9543-42f0ac7380ed","Type":"ContainerStarted","Data":"3cb9e164205e4d7dbebb1f3a01e8e05b78159b764acb6965531d900df8201287"}
Oct 30 22:15:22.578042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.578020    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"
Oct 30 22:15:22.578156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.578059    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx" event={"ID":"d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a","Type":"ContainerDied","Data":"8aadb62b8653fe5e6ea594a07d910511198a9a8c96be1d949cf59bf7da7ddb70"}
Oct 30 22:15:22.578156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.578117    5503 scope.go:122] "RemoveContainer" containerID="a16feec701ce93a7b21c38670d21651db2d278eb6689a9088e90294c17f0c6f3"
Oct 30 22:15:22.694705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.694629    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p" podStartSLOduration=1.6946165949999998 podStartE2EDuration="1.694616595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:22.693865839 +0000 UTC m=+1699.028442607" watchObservedRunningTime="2026-10-30 22:15:22.694616595 +0000 UTC m=+1699.029193365"
Oct 30 22:15:22.809201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.809113    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"]
Oct 30 22:15:22.815474 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:22.815450    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-4401bb-29889975-s8bkx"]
Oct 30 22:15:23.585748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:23.585712    5503 generic.go:356] "Generic (PLEG): container finished" podID="e73ee2dc-660d-4f05-9543-42f0ac7380ed" containerID="2d3c89194cc53faf7aa26df32cd70f3d53de2ff41afd63557758b1b30affb98c" exitCode=0
Oct 30 22:15:23.586186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:23.585801    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p" event={"ID":"e73ee2dc-660d-4f05-9543-42f0ac7380ed","Type":"ContainerDied","Data":"2d3c89194cc53faf7aa26df32cd70f3d53de2ff41afd63557758b1b30affb98c"}
Oct 30 22:15:23.790178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:23.790149    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" path="/var/lib/kubelet/pods/d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a/volumes"
Oct 30 22:15:24.752838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.752815    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:24.903101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.903063    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-xks-ca\") pod \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") "
Oct 30 22:15:24.903279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.903222    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-openshift-ca\") pod \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") "
Oct 30 22:15:24.903346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.903291    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73ee2dc-660d-4f05-9543-42f0ac7380ed-kube-api-access-z22p7\" (UniqueName: \"kubernetes.io/projected/e73ee2dc-660d-4f05-9543-42f0ac7380ed-kube-api-access-z22p7\") pod \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\" (UID: \"e73ee2dc-660d-4f05-9543-42f0ac7380ed\") "
Oct 30 22:15:24.903617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.903587    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-openshift-ca" pod "e73ee2dc-660d-4f05-9543-42f0ac7380ed" (UID: "e73ee2dc-660d-4f05-9543-42f0ac7380ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:24.904441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.904396    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:24.905818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.905779    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-xks-ca" pod "e73ee2dc-660d-4f05-9543-42f0ac7380ed" (UID: "e73ee2dc-660d-4f05-9543-42f0ac7380ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:24.906398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:24.906369    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73ee2dc-660d-4f05-9543-42f0ac7380ed-kube-api-access-z22p7" pod "e73ee2dc-660d-4f05-9543-42f0ac7380ed" (UID: "e73ee2dc-660d-4f05-9543-42f0ac7380ed"). InnerVolumeSpecName "kube-api-access-z22p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:25.005626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.005592    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73ee2dc-660d-4f05-9543-42f0ac7380ed-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.005626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.005623    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z22p7\" (UniqueName: \"kubernetes.io/projected/e73ee2dc-660d-4f05-9543-42f0ac7380ed-kube-api-access-z22p7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.534185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.534161    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:15:25.600894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.600808    5503 generic.go:356] "Generic (PLEG): container finished" podID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerID="6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd" exitCode=0
Oct 30 22:15:25.600894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.600864    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"
Oct 30 22:15:25.601094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.600891    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" event={"ID":"abc2c764-661b-4dbe-8bab-ddca18905cf3","Type":"ContainerDied","Data":"6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd"}
Oct 30 22:15:25.601094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.600972    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z" event={"ID":"abc2c764-661b-4dbe-8bab-ddca18905cf3","Type":"ContainerDied","Data":"5930f5a12f2378ee90b0adaf681b79927a17325a7e37b547924ec62eec80ff08"}
Oct 30 22:15:25.601094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.601002    5503 scope.go:122] "RemoveContainer" containerID="6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd"
Oct 30 22:15:25.602510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.602485    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p" event={"ID":"e73ee2dc-660d-4f05-9543-42f0ac7380ed","Type":"ContainerDied","Data":"3cb9e164205e4d7dbebb1f3a01e8e05b78159b764acb6965531d900df8201287"}
Oct 30 22:15:25.602510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.602513    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"
Oct 30 22:15:25.602754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.602530    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb9e164205e4d7dbebb1f3a01e8e05b78159b764acb6965531d900df8201287"
Oct 30 22:15:25.612270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.612250    5503 scope.go:122] "RemoveContainer" containerID="8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465"
Oct 30 22:15:25.621650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.621611    5503 scope.go:122] "RemoveContainer" containerID="6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd"
Oct 30 22:15:25.621909 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:25.621892    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd\": container with ID starting with 6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd not found: ID does not exist" containerID="6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd"
Oct 30 22:15:25.621972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.621918    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd"} err="failed to get container status \"6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd\": rpc error: code = NotFound desc = could not find container \"6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd\": container with ID starting with 6ea65984702b3239ad71ea844df921f1c77c67ea6cd384ce6b17b1e4e3b0a3bd not found: ID does not exist"
Oct 30 22:15:25.621972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.621933    5503 scope.go:122] "RemoveContainer" containerID="8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465"
Oct 30 22:15:25.622186 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:25.622160    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465\": container with ID starting with 8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465 not found: ID does not exist" containerID="8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465"
Oct 30 22:15:25.622264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.622189    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465"} err="failed to get container status \"8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465\": rpc error: code = NotFound desc = could not find container \"8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465\": container with ID starting with 8b477a92a8ec3766f56d53257d4b9f7b8f2a35163b05365e039b54f69b3d9465 not found: ID does not exist"
Oct 30 22:15:25.712915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.712883    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-socket\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.712998    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-credential-socket\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713047    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-kube-api-access-gqvkh\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-kube-api-access-gqvkh\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713073    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-data\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713146    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-certs\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713190    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc2c764-661b-4dbe-8bab-ddca18905cf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc2c764-661b-4dbe-8bab-ddca18905cf3-istiod-ca-cert\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713221    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-socket" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:25.713271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713230    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-envoy\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713347    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-credential-socket" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:25.713481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713350    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-podinfo\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.713481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713421    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-token\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-token\") pod \"abc2c764-661b-4dbe-8bab-ddca18905cf3\" (UID: \"abc2c764-661b-4dbe-8bab-ddca18905cf3\") "
Oct 30 22:15:25.714562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.713839    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-data" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:25.714562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.714132    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.714562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.714154    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.714562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.714168    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.714562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.714229    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-certs" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:25.714562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.714375    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc2c764-661b-4dbe-8bab-ddca18905cf3-istiod-ca-cert" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:25.715941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.715913    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-podinfo" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:25.716049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.715942    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-envoy" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:25.716049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.715974    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-token" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:25.717100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.717082    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-kube-api-access-gqvkh" pod "abc2c764-661b-4dbe-8bab-ddca18905cf3" (UID: "abc2c764-661b-4dbe-8bab-ddca18905cf3"). InnerVolumeSpecName "kube-api-access-gqvkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:25.814750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.814724    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvkh\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-kube-api-access-gqvkh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.814750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.814748    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.815099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.814760    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc2c764-661b-4dbe-8bab-ddca18905cf3-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.815099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.814769    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.815099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.814778    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:25.815099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:25.814786    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc2c764-661b-4dbe-8bab-ddca18905cf3-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:26.020740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:26.020706    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"]
Oct 30 22:15:26.026053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:26.026028    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-872730-openshift-default-666b77ccc9-b2f2z"]
Oct 30 22:15:26.725446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:26.725414    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:27.789741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:27.789709    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" path="/var/lib/kubelet/pods/abc2c764-661b-4dbe-8bab-ddca18905cf3/volumes"
Oct 30 22:15:28.099627 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:28.099537    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:15:31.467268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:31.467232    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:33.122175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:33.122142    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:33.275806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:33.275765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:36.015012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:36.014980    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:40.054687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:40.054648    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"]
Oct 30 22:15:40.105000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:40.104970    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-180aaba5-rzqmf"]
Oct 30 22:15:40.399011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:40.398935    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:40.552470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:40.552438    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:41.789075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:41.789035    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba1da2c1-0f18-4f46-982b-8b85038125da" path="/var/lib/kubelet/pods/ba1da2c1-0f18-4f46-982b-8b85038125da/volumes"
Oct 30 22:15:44.264740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:44.264706    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:15:45.262688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:45.262654    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:46.638645 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.638601    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"]
Oct 30 22:15:46.639113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639098    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:15:46.639157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639114    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:15:46.639157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639130    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:15:46.639157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639135    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:15:46.639157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639143    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy"
Oct 30 22:15:46.639157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639149    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639172    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639177    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639184    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639188    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639195    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e73ee2dc-660d-4f05-9543-42f0ac7380ed" containerName="revoke-keys"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639199    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73ee2dc-660d-4f05-9543-42f0ac7380ed" containerName="revoke-keys"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639206    5503 cpu_manager.go:414] "containerMap: removing container" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639210    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639221    5503 cpu_manager.go:414] "containerMap: removing container" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:46.639306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639225    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb2cd334-3fae-467a-ad8f-cc3b15c26b73" containerName="cleanup"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639319    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639327    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639335    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9213ee9a-5bfb-49c1-bee6-385569d0ab70" containerName="cleanup"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639342    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5973f004-7df1-498f-b740-29a227a3860c" containerName="cleanup"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639349    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73ee2dc-660d-4f05-9543-42f0ac7380ed" containerName="revoke-keys"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639355    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639363    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ae1a9c-59b0-415f-8c62-89fd4701e9f4" containerName="cleanup"
Oct 30 22:15:46.639594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.639368    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9758362-f06d-44af-8e1c-6013d5d3ab13" containerName="cleanup"
Oct 30 22:15:46.644661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.644623    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.697952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.697925    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4cf21208-openshift-default-dockercfg-js79h\""
Oct 30 22:15:46.781344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781300    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781343    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781416    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781437    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781498    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24807402-0466-48b6-b00f-010db3b9dd67-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781561    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24807402-0466-48b6-b00f-010db3b9dd67-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781587    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781761    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.781911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.781891    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkgl\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-kube-api-access-trkgl\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.842402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.842364    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"]
Oct 30 22:15:46.883201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883175    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883205    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883256    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883273    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883298    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24807402-0466-48b6-b00f-010db3b9dd67-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883336    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24807402-0466-48b6-b00f-010db3b9dd67-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883360    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883465    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883599    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkgl\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-kube-api-access-trkgl\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883623    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883657    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883846    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.883939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.883904    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.884391 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.884370    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24807402-0466-48b6-b00f-010db3b9dd67-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.885867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.885841    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24807402-0466-48b6-b00f-010db3b9dd67-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.885963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.885924    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.946026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.945993    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.946172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.946079    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkgl\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-kube-api-access-trkgl\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:46.958878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:46.958857    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:47.395885 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:15:47.395853    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24807402_0466_48b6_b00f_010db3b9dd67.slice/crio-5c2d192dffb36d5d41f3478c7b7e6005b789451e131ab358cd139460720142f5 WatchSource:0}: Error finding container 5c2d192dffb36d5d41f3478c7b7e6005b789451e131ab358cd139460720142f5: Status 404 returned error can't find the container with id 5c2d192dffb36d5d41f3478c7b7e6005b789451e131ab358cd139460720142f5
Oct 30 22:15:47.397846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.397824    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"]
Oct 30 22:15:47.398369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.398333    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:15:47.398435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.398400    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:15:47.398435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.398430    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:15:47.757595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.757557    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" event={"ID":"24807402-0466-48b6-b00f-010db3b9dd67","Type":"ContainerStarted","Data":"66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5"}
Oct 30 22:15:47.758005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.757612    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" event={"ID":"24807402-0466-48b6-b00f-010db3b9dd67","Type":"ContainerStarted","Data":"5c2d192dffb36d5d41f3478c7b7e6005b789451e131ab358cd139460720142f5"}
Oct 30 22:15:47.884403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.884331    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" podStartSLOduration=1.884316671 podStartE2EDuration="1.884316671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:47.882843893 +0000 UTC m=+1724.217420662" watchObservedRunningTime="2026-10-30 22:15:47.884316671 +0000 UTC m=+1724.218893440"
Oct 30 22:15:47.959066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:47.959034    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:48.664169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:48.664124    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:48.970090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:48.967790    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:15:48.970090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:48.967877    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 22:15:49.137944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:49.137876    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:49.959149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:49.959106    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:15:49.959351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:49.959160    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 22:15:50.959404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:50.959368    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:15:50.959837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:50.959432    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 22:15:51.561523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:51.561486    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:51.959841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:51.959802    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:15:51.960243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:51.959868    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 22:15:52.525516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:52.525356    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:52.959728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:52.959697    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 30 22:15:52.959923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:52.959773    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 30 22:15:53.768904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:53.768865    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:53.919629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:53.919593    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:54.035558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:54.035461    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:54.036054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:54.035891    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:54.036912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:54.036880    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:15:56.128388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:15:56.128350    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:15:58.099871 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:15:58.099835    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:16:00.938535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:00.938467    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:01.771464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.771430    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"]
Oct 30 22:16:01.782688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.772378    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:16:01.784072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.784045    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:16:01.784476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.784459    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc2c764-661b-4dbe-8bab-ddca18905cf3" containerName="istio-proxy"
Oct 30 22:16:01.784603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.784590    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7feb5e7-40f9-4a32-b6dc-b11e1d2b694a" containerName="cleanup"
Oct 30 22:16:01.808659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.806831    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.885842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.885803    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpcn\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-kube-api-access-kdpcn\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.886142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.886116    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.886372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.886352    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.886500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.886483    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.886612 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.886597    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5603e-96d3-4cf8-8123-e896135b277f-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.886919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.886896    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.887088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.887071    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.887335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.887315    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.887948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.887928    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.920689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.920650    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"]
Oct 30 22:16:01.989579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989543    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpcn\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-kube-api-access-kdpcn\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989599    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989698    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989723    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989739    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5603e-96d3-4cf8-8123-e896135b277f-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989789    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989824    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989921    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.989988    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.990017    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.990094    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.990182    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.990271    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.990584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.990485    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5603e-96d3-4cf8-8123-e896135b277f-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.992001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.991979    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:01.992385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:01.992365    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:02.054837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.054767    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpcn\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-kube-api-access-kdpcn\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:02.057950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.057921    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:02.170556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.170524    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:02.597384 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:16:02.597350    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac5603e_96d3_4cf8_8123_e896135b277f.slice/crio-b0ab3e54644bef1d11d55066dfca7a05c428a0ae5b6bf9c932d73d31f09553af WatchSource:0}: Error finding container b0ab3e54644bef1d11d55066dfca7a05c428a0ae5b6bf9c932d73d31f09553af: Status 404 returned error can't find the container with id b0ab3e54644bef1d11d55066dfca7a05c428a0ae5b6bf9c932d73d31f09553af
Oct 30 22:16:02.600221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.600183    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:16:02.600339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.600269    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:16:02.600339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.600314    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:16:02.600452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.600358    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"]
Oct 30 22:16:02.948417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.948375    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" event={"ID":"3ac5603e-96d3-4cf8-8123-e896135b277f","Type":"ContainerStarted","Data":"eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f"}
Oct 30 22:16:02.948622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:02.948476    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" event={"ID":"3ac5603e-96d3-4cf8-8123-e896135b277f","Type":"ContainerStarted","Data":"b0ab3e54644bef1d11d55066dfca7a05c428a0ae5b6bf9c932d73d31f09553af"}
Oct 30 22:16:03.086875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:03.086816    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podStartSLOduration=2.086799013 podStartE2EDuration="2.086799013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:03.084984737 +0000 UTC m=+1739.419561505" watchObservedRunningTime="2026-10-30 22:16:03.086799013 +0000 UTC m=+1739.421375782"
Oct 30 22:16:03.170964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:03.170935    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:03.172731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:03.172678    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:16:03.172866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:03.172741    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:16:04.171254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:04.171211    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.171770 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:04.171281    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:16:04.324790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:04.324756    5503 scope.go:122] "RemoveContainer" containerID="dadcaedd947283dafb4e0846b7d77ef84166ee16f0c11b4ac3308977ee3042ae"
Oct 30 22:16:05.171220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:05.171184    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.171382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:05.171236    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:16:05.343119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:05.343086    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:06.171334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:06.171293    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:16:06.171514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:06.171358    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:16:07.140735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.140682    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:07.170969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.170932    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.171143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.170990    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:16:07.440543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.440501    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:07.497060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.496938    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"]
Oct 30 22:16:07.497731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.497695    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy" containerID="cri-o://7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380" gracePeriod=30
Oct 30 22:16:07.596829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.596771    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:07.703877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.703758    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"]
Oct 30 22:16:07.708057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.708029    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-161111-6z4rc"]
Oct 30 22:16:07.789605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:07.789572    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3304d763-6c48-431f-b065-e66788f30a4c" path="/var/lib/kubelet/pods/3304d763-6c48-431f-b065-e66788f30a4c/volumes"
Oct 30 22:16:08.086252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:08.086160    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:08.171557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:08.171527    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.171756 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:08.171579    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:16:08.248224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:08.248187    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:08.784658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:08.776111    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:09.121801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:09.121693    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:09.171221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:09.171176    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.171427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:09.171251    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 30 22:16:10.176259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:10.176230    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:10.176763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:10.176604    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:10.177249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:10.177231    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:16:12.777410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.777343    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:16:12.824283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.824254    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887053    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-certs\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887141    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-credential-socket\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887309    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-podinfo\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887347    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-envoy\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887417    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-token\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-token\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887448    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-data\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887497    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-kube-api-access-wg928\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-kube-api-access-wg928\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887562    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istiod-ca-cert\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887589    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-socket\") pod \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\" (UID: \"27b7fd88-f60c-4e68-9de5-6180cf7ffc91\") "
Oct 30 22:16:12.887660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887650    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-credential-socket" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:12.888544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.887864    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-certs" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:12.888544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.888117    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-socket" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:12.888544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.888356    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.888544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.888378    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.888544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.888394    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.917696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.914820    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-envoy" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:12.925626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.921493    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-data" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:12.925626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.921860    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istiod-ca-cert" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:12.927683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.927619    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-token" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:12.931117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.931075    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-podinfo" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:12.939045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.939013    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-kube-api-access-wg928" pod "27b7fd88-f60c-4e68-9de5-6180cf7ffc91" (UID: "27b7fd88-f60c-4e68-9de5-6180cf7ffc91"). InnerVolumeSpecName "kube-api-access-wg928". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:12.989570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.989539    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.989570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.989572    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.989842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.989586    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.989842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.989598    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.989842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.989611    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg928\" (UniqueName: \"kubernetes.io/projected/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-kube-api-access-wg928\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.989842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:12.989626    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27b7fd88-f60c-4e68-9de5-6180cf7ffc91-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:13.097698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.097575    5503 generic.go:356] "Generic (PLEG): container finished" podID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerID="7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380" exitCode=0
Oct 30 22:16:13.097698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.097624    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"
Oct 30 22:16:13.097698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.097662    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" event={"ID":"27b7fd88-f60c-4e68-9de5-6180cf7ffc91","Type":"ContainerDied","Data":"7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380"}
Oct 30 22:16:13.097976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.097741    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf" event={"ID":"27b7fd88-f60c-4e68-9de5-6180cf7ffc91","Type":"ContainerDied","Data":"97dae7c40af1c1b454815112f51a2808ff16c2c6639887506cd29925acab43d5"}
Oct 30 22:16:13.097976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.097772    5503 scope.go:122] "RemoveContainer" containerID="7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380"
Oct 30 22:16:13.107569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.107551    5503 scope.go:122] "RemoveContainer" containerID="7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380"
Oct 30 22:16:13.107877 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:16:13.107859    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380\": container with ID starting with 7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380 not found: ID does not exist" containerID="7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380"
Oct 30 22:16:13.107939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.107886    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380"} err="failed to get container status \"7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380\": rpc error: code = NotFound desc = could not find container \"7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380\": container with ID starting with 7fca4e228fc7a7c18c4f99cb06d1cd6b46232a1a6d549ad3a07c15a7aa5ac380 not found: ID does not exist"
Oct 30 22:16:13.267085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.267051    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"]
Oct 30 22:16:13.275735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.275704    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-6w4sf"]
Oct 30 22:16:13.462909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.462875    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:13.626299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.626269    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:13.789184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:13.789104    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" path="/var/lib/kubelet/pods/27b7fd88-f60c-4e68-9de5-6180cf7ffc91/volumes"
Oct 30 22:16:14.592486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.592457    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"]
Oct 30 22:16:14.593000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.592987    5503 cpu_manager.go:414] "containerMap: removing container" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy"
Oct 30 22:16:14.593057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.593002    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy"
Oct 30 22:16:14.593094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.593073    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b7fd88-f60c-4e68-9de5-6180cf7ffc91" containerName="istio-proxy"
Oct 30 22:16:14.597693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.597676    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.651553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.651419    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"]
Oct 30 22:16:14.698713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.698687    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:14.698846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.698731    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:16:14.698886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.698847    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:16:14.702062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.702049    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:16:14.710793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.710768    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef1151b-5089-4c97-b231-618362103767-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.710894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.710813    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1151b-5089-4c97-b231-618362103767-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.710933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.710908    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnt8d\" (UniqueName: \"kubernetes.io/projected/6ef1151b-5089-4c97-b231-618362103767-kube-api-access-nnt8d\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.812282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.812255    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef1151b-5089-4c97-b231-618362103767-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.812738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.812301    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1151b-5089-4c97-b231-618362103767-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.812738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.812342    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnt8d\" (UniqueName: \"kubernetes.io/projected/6ef1151b-5089-4c97-b231-618362103767-kube-api-access-nnt8d\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.813359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.813330    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1151b-5089-4c97-b231-618362103767-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.814867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.814840    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef1151b-5089-4c97-b231-618362103767-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.854086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.854010    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:14.874552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.874522    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnt8d\" (UniqueName: \"kubernetes.io/projected/6ef1151b-5089-4c97-b231-618362103767-kube-api-access-nnt8d\") pod \"maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:14.921133 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:14.921097    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:15.019372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:15.019325    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:15.163574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:15.163544    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"]
Oct 30 22:16:15.163945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:15.163793    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy" containerID="cri-o://130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54" gracePeriod=30
Oct 30 22:16:15.211298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:15.207761    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:15.405677 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:16:15.398332    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef1151b_5089_4c97_b231_618362103767.slice/crio-6abe750e275fbe6b993e24672511b834f0498407c53c17f07f91ae5f3dc688ac WatchSource:0}: Error finding container 6abe750e275fbe6b993e24672511b834f0498407c53c17f07f91ae5f3dc688ac: Status 404 returned error can't find the container with id 6abe750e275fbe6b993e24672511b834f0498407c53c17f07f91ae5f3dc688ac
Oct 30 22:16:15.422337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:15.422249    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"]
Oct 30 22:16:16.118363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.118282    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h" event={"ID":"6ef1151b-5089-4c97-b231-618362103767","Type":"ContainerStarted","Data":"08fcd5590ef4fe32053365a470a606445f2086bc7299f27484d4d6d84c39ba2d"}
Oct 30 22:16:16.118363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.118334    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h" event={"ID":"6ef1151b-5089-4c97-b231-618362103767","Type":"ContainerStarted","Data":"6abe750e275fbe6b993e24672511b834f0498407c53c17f07f91ae5f3dc688ac"}
Oct 30 22:16:16.241977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.241916    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h" podStartSLOduration=2.241897958 podStartE2EDuration="2.241897958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:16.239838053 +0000 UTC m=+1752.574414825" watchObservedRunningTime="2026-10-30 22:16:16.241897958 +0000 UTC m=+1752.576474731"
Oct 30 22:16:16.861885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.861852    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"]
Oct 30 22:16:16.865840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.865817    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.885896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.885869    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.885911    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be720949-f679-4113-bb9e-3bc80e3c4cf1-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.885943    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-kube-api-access-mp86h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.886067    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.886118    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.886210    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.886245    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.886271    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.886360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.886314    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.961048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.961016    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"]
Oct 30 22:16:16.987201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987175    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987216    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be720949-f679-4113-bb9e-3bc80e3c4cf1-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987250    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-kube-api-access-mp86h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987279    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987299    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987334    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987368    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987407    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987432    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.987939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.987892    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.988112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.988005    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-certs\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.988112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.988027    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-data\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.988309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.988162    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be720949-f679-4113-bb9e-3bc80e3c4cf1-istiod-ca-cert\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.988401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.988348    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-credential-socket\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.990316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.990280    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-envoy\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:16.990669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:16.990626    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-podinfo\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:17.049984 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.049956    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-token\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:17.050157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.050037    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-kube-api-access-mp86h\") pod \"e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") " pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:17.126622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.126533    5503 generic.go:356] "Generic (PLEG): container finished" podID="6ef1151b-5089-4c97-b231-618362103767" containerID="08fcd5590ef4fe32053365a470a606445f2086bc7299f27484d4d6d84c39ba2d" exitCode=0
Oct 30 22:16:17.127031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.126618    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h" event={"ID":"6ef1151b-5089-4c97-b231-618362103767","Type":"ContainerDied","Data":"08fcd5590ef4fe32053365a470a606445f2086bc7299f27484d4d6d84c39ba2d"}
Oct 30 22:16:17.179891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.179854    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:17.410712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.404732    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:17.609658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.599741    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"]
Oct 30 22:16:17.625657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.618244    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.652654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.650481    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"]
Oct 30 22:16:17.688884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.688811    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"]
Oct 30 22:16:17.713346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.713306    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.713458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.713425    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.713566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.713549    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6b8\" (UniqueName: \"kubernetes.io/projected/5934ca8e-788f-42d1-9384-646deb04c0ef-kube-api-access-bc6b8\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.714413 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:16:17.714391    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe720949_f679_4113_bb9e_3bc80e3c4cf1.slice/crio-d5f0dc4884bbdb4398ae9cff50f983d8d2b116e4fcb516ba37c9976f659061c1 WatchSource:0}: Error finding container d5f0dc4884bbdb4398ae9cff50f983d8d2b116e4fcb516ba37c9976f659061c1: Status 404 returned error can't find the container with id d5f0dc4884bbdb4398ae9cff50f983d8d2b116e4fcb516ba37c9976f659061c1
Oct 30 22:16:17.716824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.716794    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:16:17.716891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.716863    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:16:17.716927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.716889    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:16:17.822989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.822952    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6b8\" (UniqueName: \"kubernetes.io/projected/5934ca8e-788f-42d1-9384-646deb04c0ef-kube-api-access-bc6b8\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.823880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.823167    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.823880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.823223    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.824408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.824379    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.831082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.830803    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:17.990075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:17.990002    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6b8\" (UniqueName: \"kubernetes.io/projected/5934ca8e-788f-42d1-9384-646deb04c0ef-kube-api-access-bc6b8\") pod \"maas-api-revoke-keys-e2e-api-b-427d70-wzjr9\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:18.135190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.135142    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" event={"ID":"be720949-f679-4113-bb9e-3bc80e3c4cf1","Type":"ContainerStarted","Data":"e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265"}
Oct 30 22:16:18.135761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.135210    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" event={"ID":"be720949-f679-4113-bb9e-3bc80e3c4cf1","Type":"ContainerStarted","Data":"d5f0dc4884bbdb4398ae9cff50f983d8d2b116e4fcb516ba37c9976f659061c1"}
Oct 30 22:16:18.180874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.180842    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:18.182598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.182562    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:18.182741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.182614    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 22:16:18.263150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.263097    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podStartSLOduration=2.263082279 podStartE2EDuration="2.263082279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:18.261136793 +0000 UTC m=+1754.595713563" watchObservedRunningTime="2026-10-30 22:16:18.263082279 +0000 UTC m=+1754.597659048"
Oct 30 22:16:18.267259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.267230    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:18.306866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.306840    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:18.369859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.369825    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef1151b-5089-4c97-b231-618362103767-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1151b-5089-4c97-b231-618362103767-maas-api-openshift-ca\") pod \"6ef1151b-5089-4c97-b231-618362103767\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") "
Oct 30 22:16:18.370039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.370017    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef1151b-5089-4c97-b231-618362103767-kube-api-access-nnt8d\" (UniqueName: \"kubernetes.io/projected/6ef1151b-5089-4c97-b231-618362103767-kube-api-access-nnt8d\") pod \"6ef1151b-5089-4c97-b231-618362103767\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") "
Oct 30 22:16:18.370104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.370077    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef1151b-5089-4c97-b231-618362103767-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef1151b-5089-4c97-b231-618362103767-maas-api-xks-ca\") pod \"6ef1151b-5089-4c97-b231-618362103767\" (UID: \"6ef1151b-5089-4c97-b231-618362103767\") "
Oct 30 22:16:18.371864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.371829    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef1151b-5089-4c97-b231-618362103767-maas-api-openshift-ca" pod "6ef1151b-5089-4c97-b231-618362103767" (UID: "6ef1151b-5089-4c97-b231-618362103767"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:18.374211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.374183    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef1151b-5089-4c97-b231-618362103767-maas-api-xks-ca" pod "6ef1151b-5089-4c97-b231-618362103767" (UID: "6ef1151b-5089-4c97-b231-618362103767"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:18.374338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.374320    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef1151b-5089-4c97-b231-618362103767-kube-api-access-nnt8d" pod "6ef1151b-5089-4c97-b231-618362103767" (UID: "6ef1151b-5089-4c97-b231-618362103767"). InnerVolumeSpecName "kube-api-access-nnt8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:18.472153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.472119    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnt8d\" (UniqueName: \"kubernetes.io/projected/6ef1151b-5089-4c97-b231-618362103767-kube-api-access-nnt8d\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:18.472153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.472151    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef1151b-5089-4c97-b231-618362103767-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:18.472346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.472162    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef1151b-5089-4c97-b231-618362103767-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:18.642977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.642895    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:18.646285 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:16:18.646253    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5934ca8e_788f_42d1_9384_646deb04c0ef.slice/crio-55727d4d9ec75db0a8424167d3b227819a976d0bc35bc399a4dd2c0d6deea77d WatchSource:0}: Error finding container 55727d4d9ec75db0a8424167d3b227819a976d0bc35bc399a4dd2c0d6deea77d: Status 404 returned error can't find the container with id 55727d4d9ec75db0a8424167d3b227819a976d0bc35bc399a4dd2c0d6deea77d
Oct 30 22:16:18.646795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:18.646772    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"]
Oct 30 22:16:19.143490 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.143444    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9" event={"ID":"5934ca8e-788f-42d1-9384-646deb04c0ef","Type":"ContainerStarted","Data":"6e409f10c09f2ed1f933fb25969b80b1355f2fc48826405f062fd39d117ca2b0"}
Oct 30 22:16:19.143997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.143505    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9" event={"ID":"5934ca8e-788f-42d1-9384-646deb04c0ef","Type":"ContainerStarted","Data":"55727d4d9ec75db0a8424167d3b227819a976d0bc35bc399a4dd2c0d6deea77d"}
Oct 30 22:16:19.145009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.144986    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"
Oct 30 22:16:19.145133 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.145022    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h" event={"ID":"6ef1151b-5089-4c97-b231-618362103767","Type":"ContainerDied","Data":"6abe750e275fbe6b993e24672511b834f0498407c53c17f07f91ae5f3dc688ac"}
Oct 30 22:16:19.145133 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.145063    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6abe750e275fbe6b993e24672511b834f0498407c53c17f07f91ae5f3dc688ac"
Oct 30 22:16:19.181700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.181549    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:19.181700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.181621    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 22:16:19.263111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.263056    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9" podStartSLOduration=2.263042619 podStartE2EDuration="2.263042619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:19.261051643 +0000 UTC m=+1755.595628411" watchObservedRunningTime="2026-10-30 22:16:19.263042619 +0000 UTC m=+1755.597619388"
Oct 30 22:16:19.439129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:19.439018    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:20.153561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.153526    5503 generic.go:356] "Generic (PLEG): container finished" podID="5934ca8e-788f-42d1-9384-646deb04c0ef" containerID="6e409f10c09f2ed1f933fb25969b80b1355f2fc48826405f062fd39d117ca2b0" exitCode=0
Oct 30 22:16:20.154022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.153616    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9" event={"ID":"5934ca8e-788f-42d1-9384-646deb04c0ef","Type":"ContainerDied","Data":"6e409f10c09f2ed1f933fb25969b80b1355f2fc48826405f062fd39d117ca2b0"}
Oct 30 22:16:20.180703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.180679    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:20.180828 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.180729    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 22:16:20.474329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.474301    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:16:20.541379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.541347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:20.608693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.608660    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-credential-socket\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.608869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.608703    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-data\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.608869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.608740    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-socket\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.608869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.608775    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-envoy\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.609041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.608902    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b132c2-4393-49be-9adb-dd50b45d2322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b132c2-4393-49be-9adb-dd50b45d2322-istiod-ca-cert\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.609041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.608943    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-credential-socket" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:20.609041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.608980    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-kube-api-access-k6db7\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-kube-api-access-k6db7\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.609041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609016    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-podinfo\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.609249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609049    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-token\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-token\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.609249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609136    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-certs\") pod \"e2b132c2-4393-49be-9adb-dd50b45d2322\" (UID: \"e2b132c2-4393-49be-9adb-dd50b45d2322\") "
Oct 30 22:16:20.609465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609439    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-certs" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:20.609556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609534    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b132c2-4393-49be-9adb-dd50b45d2322-istiod-ca-cert" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:20.609780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609755    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.609971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609951    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.609971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609972    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b132c2-4393-49be-9adb-dd50b45d2322-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.610147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609787    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-socket" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:20.610147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.609828    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-data" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:20.611564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.611539    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-envoy" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:20.611887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.611841    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-podinfo" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:20.612020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.611996    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-token" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:20.612081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.612040    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-kube-api-access-k6db7" pod "e2b132c2-4393-49be-9adb-dd50b45d2322" (UID: "e2b132c2-4393-49be-9adb-dd50b45d2322"). InnerVolumeSpecName "kube-api-access-k6db7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:20.711438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.711347    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6db7\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-kube-api-access-k6db7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.711438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.711387    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.711438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.711401    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.711438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.711416    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.711438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.711431    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:20.711438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:20.711445    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b132c2-4393-49be-9adb-dd50b45d2322-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:21.161114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.161081    5503 generic.go:356] "Generic (PLEG): container finished" podID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerID="130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54" exitCode=0
Oct 30 22:16:21.161516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.161143    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"
Oct 30 22:16:21.161516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.161159    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" event={"ID":"e2b132c2-4393-49be-9adb-dd50b45d2322","Type":"ContainerDied","Data":"130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54"}
Oct 30 22:16:21.161516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.161218    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q" event={"ID":"e2b132c2-4393-49be-9adb-dd50b45d2322","Type":"ContainerDied","Data":"b709c96c9f83f221048639d7c7ce5ef0afb41d9fb35a983a6e7c95cc6256e43a"}
Oct 30 22:16:21.161516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.161241    5503 scope.go:122] "RemoveContainer" containerID="130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54"
Oct 30 22:16:21.172485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.172465    5503 scope.go:122] "RemoveContainer" containerID="fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b"
Oct 30 22:16:21.180841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.180814    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:21.180964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.180857    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 22:16:21.182434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.182417    5503 scope.go:122] "RemoveContainer" containerID="130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54"
Oct 30 22:16:21.182746 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:16:21.182715    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54\": container with ID starting with 130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54 not found: ID does not exist" containerID="130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54"
Oct 30 22:16:21.182829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.182756    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54"} err="failed to get container status \"130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54\": rpc error: code = NotFound desc = could not find container \"130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54\": container with ID starting with 130cc03c3458c0030bc069d01b968e4bac3b9aa5fe8f304b1bd9d9ea9a61ff54 not found: ID does not exist"
Oct 30 22:16:21.182829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.182776    5503 scope.go:122] "RemoveContainer" containerID="fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b"
Oct 30 22:16:21.183051 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:16:21.183027    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b\": container with ID starting with fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b not found: ID does not exist" containerID="fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b"
Oct 30 22:16:21.183113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.183056    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b"} err="failed to get container status \"fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b\": rpc error: code = NotFound desc = could not find container \"fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b\": container with ID starting with fbc7cd91294a32abef4f565cd92ae19d30174613471f31a6301854565de5ee6b not found: ID does not exist"
Oct 30 22:16:21.356082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.355947    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:21.363750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.363726    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"]
Oct 30 22:16:21.440746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.439058    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-161111-openshift-default-cf8877bd7-tq97q"]
Oct 30 22:16:21.446017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.445984    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5934ca8e-788f-42d1-9384-646deb04c0ef-kube-api-access-bc6b8\" (UniqueName: \"kubernetes.io/projected/5934ca8e-788f-42d1-9384-646deb04c0ef-kube-api-access-bc6b8\") pod \"5934ca8e-788f-42d1-9384-646deb04c0ef\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") "
Oct 30 22:16:21.446611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.446584    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-xks-ca\") pod \"5934ca8e-788f-42d1-9384-646deb04c0ef\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") "
Oct 30 22:16:21.446742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.446681    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-openshift-ca\") pod \"5934ca8e-788f-42d1-9384-646deb04c0ef\" (UID: \"5934ca8e-788f-42d1-9384-646deb04c0ef\") "
Oct 30 22:16:21.447066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.447040    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-openshift-ca" pod "5934ca8e-788f-42d1-9384-646deb04c0ef" (UID: "5934ca8e-788f-42d1-9384-646deb04c0ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:21.447835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.447816    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:21.448913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.448889    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-xks-ca" pod "5934ca8e-788f-42d1-9384-646deb04c0ef" (UID: "5934ca8e-788f-42d1-9384-646deb04c0ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:21.449006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.448978    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5934ca8e-788f-42d1-9384-646deb04c0ef-kube-api-access-bc6b8" pod "5934ca8e-788f-42d1-9384-646deb04c0ef" (UID: "5934ca8e-788f-42d1-9384-646deb04c0ef"). InnerVolumeSpecName "kube-api-access-bc6b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:21.550930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.550892    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5934ca8e-788f-42d1-9384-646deb04c0ef-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:21.551099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.550935    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6b8\" (UniqueName: \"kubernetes.io/projected/5934ca8e-788f-42d1-9384-646deb04c0ef-kube-api-access-bc6b8\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:21.789897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:21.789818    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" path="/var/lib/kubelet/pods/e2b132c2-4393-49be-9adb-dd50b45d2322/volumes"
Oct 30 22:16:22.170345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:22.170319    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"
Oct 30 22:16:22.170784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:22.170340    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9" event={"ID":"5934ca8e-788f-42d1-9384-646deb04c0ef","Type":"ContainerDied","Data":"55727d4d9ec75db0a8424167d3b227819a976d0bc35bc399a4dd2c0d6deea77d"}
Oct 30 22:16:22.170784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:22.170394    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55727d4d9ec75db0a8424167d3b227819a976d0bc35bc399a4dd2c0d6deea77d"
Oct 30 22:16:22.180761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:22.180735    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:22.180884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:22.180779    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 22:16:22.866656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:22.865391    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:23.181025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:23.180994    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:23.181400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:23.181042    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 22:16:23.934825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:23.934786    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:16:24.180159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:24.180125    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:16:24.180337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:24.180197    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 22:16:25.184842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:25.184816    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:25.185248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:25.185226    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:25.185829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:25.185808    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:16:26.523879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:26.523849    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:26.703932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:26.703898    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:27.608977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:27.608942    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:28.196567 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:16:28.196526    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:16:29.253295 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:29.253254    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:29.409078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:29.409041    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:32.228665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:32.228614    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:36.921134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:36.921100    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:37.701815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:37.701778    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:38.635480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:38.635448    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.102134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:39.102092    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.258928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:39.258891    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.433670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:39.433242    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:39.592081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:39.592039    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:43.370031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:43.370001    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:44.216786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:44.216746    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:44.376086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:44.376051    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:44.543144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:44.543037    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:47.585238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:47.585200    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:47.818592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:47.809943    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:47.972329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:47.972293    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:49.571890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:49.571856    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:49.814139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:49.814094    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:49.976844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:49.976808    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:50.016118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:50.016084    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"]
Oct 30 22:16:50.066922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:50.066889    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4cf21208-cgm9h"]
Oct 30 22:16:50.476578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:50.476535    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:51.789966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:51.789934    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef1151b-5089-4c97-b231-618362103767" path="/var/lib/kubelet/pods/6ef1151b-5089-4c97-b231-618362103767/volumes"
Oct 30 22:16:54.753070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:54.753037    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:54.911154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:54.911123    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:55.097017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:55.096936    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:55.429074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:55.429041    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"]
Oct 30 22:16:55.525040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:55.525010    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:16:58.098743 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:16:58.098712    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:16:59.625527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:16:59.625491    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:03.587839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:03.587805    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:04.018203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:04.018171    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:17:04.057460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:04.057428    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:17:04.434674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:04.423718    5503 scope.go:122] "RemoveContainer" containerID="4d7a51a89b6887350c56c4bac8b13c4bd0fce1c5380698e2fec0e6108a254371"
Oct 30 22:17:05.008204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:05.008165    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.228297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:05.228263    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.383905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:05.383788    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:05.478388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:05.478354    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"]
Oct 30 22:17:05.529265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:05.529234    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4401bb-m299p"]
Oct 30 22:17:05.789047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:05.789014    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73ee2dc-660d-4f05-9543-42f0ac7380ed" path="/var/lib/kubelet/pods/e73ee2dc-660d-4f05-9543-42f0ac7380ed/volumes"
Oct 30 22:17:05.910565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:05.910524    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:09.916782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:09.916738    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:10.450305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.450260    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7" podUID="8fa83512-318e-4a91-82cd-25a1404dabcd" containerName="main" containerID="cri-o://d2a443169df79b9a084e5b8874ab890c87aabb29fa9a84bbe194e50ac28041b5" gracePeriod=45
Oct 30 22:17:10.600760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.600721    5503 generic.go:356] "Generic (PLEG): container finished" podID="8fa83512-318e-4a91-82cd-25a1404dabcd" containerID="d2a443169df79b9a084e5b8874ab890c87aabb29fa9a84bbe194e50ac28041b5" exitCode=0
Oct 30 22:17:10.600935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.600792    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7" event={"ID":"8fa83512-318e-4a91-82cd-25a1404dabcd","Type":"ContainerDied","Data":"d2a443169df79b9a084e5b8874ab890c87aabb29fa9a84bbe194e50ac28041b5"}
Oct 30 22:17:10.711107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.711050    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:17:10.782544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.782516    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-model-cache\") pod \"8fa83512-318e-4a91-82cd-25a1404dabcd\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") "
Oct 30 22:17:10.782733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.782595    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa83512-318e-4a91-82cd-25a1404dabcd-kube-api-access-47cn4\" (UniqueName: \"kubernetes.io/projected/8fa83512-318e-4a91-82cd-25a1404dabcd-kube-api-access-47cn4\") pod \"8fa83512-318e-4a91-82cd-25a1404dabcd\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") "
Oct 30 22:17:10.782733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.782627    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa83512-318e-4a91-82cd-25a1404dabcd-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa83512-318e-4a91-82cd-25a1404dabcd-tls-certs\") pod \"8fa83512-318e-4a91-82cd-25a1404dabcd\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") "
Oct 30 22:17:10.782733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.782694    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-dshm\") pod \"8fa83512-318e-4a91-82cd-25a1404dabcd\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") "
Oct 30 22:17:10.782901 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.782750    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-tmp-dir\") pod \"8fa83512-318e-4a91-82cd-25a1404dabcd\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") "
Oct 30 22:17:10.782901 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.782771    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-home\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-home\") pod \"8fa83512-318e-4a91-82cd-25a1404dabcd\" (UID: \"8fa83512-318e-4a91-82cd-25a1404dabcd\") "
Oct 30 22:17:10.782901 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.782826    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-model-cache" pod "8fa83512-318e-4a91-82cd-25a1404dabcd" (UID: "8fa83512-318e-4a91-82cd-25a1404dabcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:10.783061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.783033    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-tmp-dir" pod "8fa83512-318e-4a91-82cd-25a1404dabcd" (UID: "8fa83512-318e-4a91-82cd-25a1404dabcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:10.783124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.783106    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-home" pod "8fa83512-318e-4a91-82cd-25a1404dabcd" (UID: "8fa83512-318e-4a91-82cd-25a1404dabcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:10.783406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.783384    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:10.783459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.783413    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:10.783459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.783431    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:10.784895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.784865    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-dshm" pod "8fa83512-318e-4a91-82cd-25a1404dabcd" (UID: "8fa83512-318e-4a91-82cd-25a1404dabcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:10.785002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.784921    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa83512-318e-4a91-82cd-25a1404dabcd-kube-api-access-47cn4" pod "8fa83512-318e-4a91-82cd-25a1404dabcd" (UID: "8fa83512-318e-4a91-82cd-25a1404dabcd"). InnerVolumeSpecName "kube-api-access-47cn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:10.785002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.784949    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa83512-318e-4a91-82cd-25a1404dabcd-tls-certs" pod "8fa83512-318e-4a91-82cd-25a1404dabcd" (UID: "8fa83512-318e-4a91-82cd-25a1404dabcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:10.885063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.885019    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cn4\" (UniqueName: \"kubernetes.io/projected/8fa83512-318e-4a91-82cd-25a1404dabcd-kube-api-access-47cn4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:10.885063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.885052    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa83512-318e-4a91-82cd-25a1404dabcd-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:10.885063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:10.885062    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa83512-318e-4a91-82cd-25a1404dabcd-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:11.609526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:11.609490    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7" event={"ID":"8fa83512-318e-4a91-82cd-25a1404dabcd","Type":"ContainerDied","Data":"1567d66bf8201f8255462452cb53cebbf9936911cd7a1d9eb0160af4b72864eb"}
Oct 30 22:17:11.610026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:11.609540    5503 scope.go:122] "RemoveContainer" containerID="d2a443169df79b9a084e5b8874ab890c87aabb29fa9a84bbe194e50ac28041b5"
Oct 30 22:17:11.610026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:11.609507    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"
Oct 30 22:17:11.779838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:11.779799    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"]
Oct 30 22:17:11.829779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:11.829744    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4cf21208/e2e-lifecycle-model-4cf21208-kserve-67cccc79c-9pmz7"]
Oct 30 22:17:12.046475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:12.046445    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.050442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:12.050412    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"]
Oct 30 22:17:12.055846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:12.055824    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52f0c9fa-bjrjm"]
Oct 30 22:17:12.202723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:12.202691    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:12.390025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:12.389945    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:13.369399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:13.369310    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:13.796950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:13.796916    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa83512-318e-4a91-82cd-25a1404dabcd" path="/var/lib/kubelet/pods/8fa83512-318e-4a91-82cd-25a1404dabcd/volumes"
Oct 30 22:17:13.797878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:13.797853    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c5bbfb-537d-4d41-a2c7-b4fab648f74b" path="/var/lib/kubelet/pods/d5c5bbfb-537d-4d41-a2c7-b4fab648f74b/volumes"
Oct 30 22:17:14.052715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:14.041141    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:14.204340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:14.204307    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:14.469003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:14.468970    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:16.023517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:16.023483    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:17.744727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:17.744695    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:17:18.477187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:18.477144    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:18.651261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:18.651226    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:18.822724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:18.822611    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:19.496658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:19.496611    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:19.834810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:19.834723    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:20.881238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:20.881203    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:21.704246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:21.704214    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:21.927268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:21.922917    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:23.294376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:23.294322    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:23.946454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:23.946415    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:26.629500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:26.629438    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:28.100615 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:17:28.100578    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:17:31.990067 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:31.990028    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:32.696495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:32.696461    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:32.867229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:32.867200    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:33.048592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:33.048508    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:33.324918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:33.324830    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:37.519182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:37.519146    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:39.449564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.449528    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:39.615294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.615261    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:39.802525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.802432    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"]
Oct 30 22:17:39.802713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.802619    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy" containerID="cri-o://66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5" gracePeriod=30
Oct 30 22:17:39.822364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.822332    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"]
Oct 30 22:17:39.822582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.822554    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" containerID="cri-o://e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265" gracePeriod=30
Oct 30 22:17:39.825189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.825140    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:39.845468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.845445    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"]
Oct 30 22:17:39.845654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:39.845621    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" containerID="cri-o://eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f" gracePeriod=30
Oct 30 22:17:40.178343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:40.178305    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:40.178525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:40.178372    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:40.187040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:40.187009    5503 patch_prober.go:29] interesting pod/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:40.187161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:40.187070    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:40.336973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:40.336935    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:40.502069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:40.501990    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:41.994199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:41.994168    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:42.150873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:42.150845    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:42.310352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:42.310266    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:45.117265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.117230    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:17:45.156582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.156549    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:17:45.160033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.160014    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:17:45.203928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.203902    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-kube-api-access-mp86h\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204048    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-credential-socket\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204102    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-podinfo\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204130    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-data\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204162    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be720949-f679-4113-bb9e-3bc80e3c4cf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be720949-f679-4113-bb9e-3bc80e3c4cf1-istiod-ca-cert\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204223    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-token\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-token\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204272    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-certs\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204347    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-envoy\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204357    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-credential-socket" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.204475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204388    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-socket\") pod \"be720949-f679-4113-bb9e-3bc80e3c4cf1\" (UID: \"be720949-f679-4113-bb9e-3bc80e3c4cf1\") "
Oct 30 22:17:45.204475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204464    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-data" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.204666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204617    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-certs" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.204848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204820    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be720949-f679-4113-bb9e-3bc80e3c4cf1-istiod-ca-cert" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:45.204934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.204910    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-socket" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.205100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.205074    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.205205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.205102    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.205205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.205120    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.205205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.205134    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.205205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.205150    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be720949-f679-4113-bb9e-3bc80e3c4cf1-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.206332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.206312    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-podinfo" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:45.206456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.206432    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-envoy" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.206820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.206806    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-kube-api-access-mp86h" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "kube-api-access-mp86h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:45.206896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.206826    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-token" pod "be720949-f679-4113-bb9e-3bc80e3c4cf1" (UID: "be720949-f679-4113-bb9e-3bc80e3c4cf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:45.306522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306492    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-certs\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.306703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306544    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24807402-0466-48b6-b00f-010db3b9dd67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24807402-0466-48b6-b00f-010db3b9dd67-istio-podinfo\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.306703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306571    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-token\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-token\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.306703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306612    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-socket\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.306703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306667    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-credential-socket\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.306703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306697    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-socket\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.306878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306736    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-credential-socket\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.306878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306785    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-certs\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.306878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306822    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-data\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.306878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306819    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-certs" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.306878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306858    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-kube-api-access-trkgl\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-kube-api-access-trkgl\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.307123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306906    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-kube-api-access-kdpcn\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-kube-api-access-kdpcn\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.307123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.306965    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24807402-0466-48b6-b00f-010db3b9dd67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24807402-0466-48b6-b00f-010db3b9dd67-istiod-ca-cert\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.307123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307019    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-socket" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.307123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307035    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-podinfo\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.307123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307069    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-istio-token\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-istio-token\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.307123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307108    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac5603e-96d3-4cf8-8123-e896135b277f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5603e-96d3-4cf8-8123-e896135b277f-istiod-ca-cert\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.307436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307151    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-envoy\") pod \"3ac5603e-96d3-4cf8-8123-e896135b277f\" (UID: \"3ac5603e-96d3-4cf8-8123-e896135b277f\") "
Oct 30 22:17:45.307436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307278    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-envoy\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.307436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307309    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-data\") pod \"24807402-0466-48b6-b00f-010db3b9dd67\" (UID: \"24807402-0466-48b6-b00f-010db3b9dd67\") "
Oct 30 22:17:45.307436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307148    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-data" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.307436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307157    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-socket" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.307436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307219    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-credential-socket" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.307886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307332    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-credential-socket" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.307886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.307412    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-certs" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308019    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac5603e-96d3-4cf8-8123-e896135b277f-istiod-ca-cert" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308045    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-data" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308178    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308196    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308210    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308225    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308238    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308252    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308266    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308279    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308291    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308305    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308319    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/be720949-f679-4113-bb9e-3bc80e3c4cf1-kube-api-access-mp86h\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308334    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac5603e-96d3-4cf8-8123-e896135b277f-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.308372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308348    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be720949-f679-4113-bb9e-3bc80e3c4cf1-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.309224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.308702    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24807402-0466-48b6-b00f-010db3b9dd67-istiod-ca-cert" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:45.309280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.309240    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24807402-0466-48b6-b00f-010db3b9dd67-istio-podinfo" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:45.309615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.309588    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-podinfo" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:45.310603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.310583    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-kube-api-access-kdpcn" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "kube-api-access-kdpcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:45.310700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.310605    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-kube-api-access-trkgl" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "kube-api-access-trkgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:45.311026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.311009    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-envoy" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.311900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.311882    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-istio-token" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:45.312143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.312130    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-envoy" pod "24807402-0466-48b6-b00f-010db3b9dd67" (UID: "24807402-0466-48b6-b00f-010db3b9dd67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:45.312779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.312761    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-token" pod "3ac5603e-96d3-4cf8-8123-e896135b277f" (UID: "3ac5603e-96d3-4cf8-8123-e896135b277f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:45.409477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409440    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24807402-0466-48b6-b00f-010db3b9dd67-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409479    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409492    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409504    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409517    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24807402-0466-48b6-b00f-010db3b9dd67-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409528    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24807402-0466-48b6-b00f-010db3b9dd67-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409540    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409569    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trkgl\" (UniqueName: \"kubernetes.io/projected/24807402-0466-48b6-b00f-010db3b9dd67-kube-api-access-trkgl\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.409743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.409584    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpcn\" (UniqueName: \"kubernetes.io/projected/3ac5603e-96d3-4cf8-8123-e896135b277f-kube-api-access-kdpcn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:45.853398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.853364    5503 generic.go:356] "Generic (PLEG): container finished" podID="24807402-0466-48b6-b00f-010db3b9dd67" containerID="66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5" exitCode=0
Oct 30 22:17:45.853529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.853416    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"
Oct 30 22:17:45.853529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.853443    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" event={"ID":"24807402-0466-48b6-b00f-010db3b9dd67","Type":"ContainerDied","Data":"66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5"}
Oct 30 22:17:45.853529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.853496    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t" event={"ID":"24807402-0466-48b6-b00f-010db3b9dd67","Type":"ContainerDied","Data":"5c2d192dffb36d5d41f3478c7b7e6005b789451e131ab358cd139460720142f5"}
Oct 30 22:17:45.853529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.853522    5503 scope.go:122] "RemoveContainer" containerID="66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5"
Oct 30 22:17:45.855116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.855095    5503 generic.go:356] "Generic (PLEG): container finished" podID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerID="e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265" exitCode=0
Oct 30 22:17:45.855235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.855142    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"
Oct 30 22:17:45.855235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.855190    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" event={"ID":"be720949-f679-4113-bb9e-3bc80e3c4cf1","Type":"ContainerDied","Data":"e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265"}
Oct 30 22:17:45.855354 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.855264    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp" event={"ID":"be720949-f679-4113-bb9e-3bc80e3c4cf1","Type":"ContainerDied","Data":"d5f0dc4884bbdb4398ae9cff50f983d8d2b116e4fcb516ba37c9976f659061c1"}
Oct 30 22:17:45.857025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.857003    5503 generic.go:356] "Generic (PLEG): container finished" podID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerID="eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f" exitCode=0
Oct 30 22:17:45.857134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.857069    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" event={"ID":"3ac5603e-96d3-4cf8-8123-e896135b277f","Type":"ContainerDied","Data":"eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f"}
Oct 30 22:17:45.857134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.857107    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj" event={"ID":"3ac5603e-96d3-4cf8-8123-e896135b277f","Type":"ContainerDied","Data":"b0ab3e54644bef1d11d55066dfca7a05c428a0ae5b6bf9c932d73d31f09553af"}
Oct 30 22:17:45.857245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.857160    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"
Oct 30 22:17:45.867403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.867373    5503 scope.go:122] "RemoveContainer" containerID="66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5"
Oct 30 22:17:45.867699 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:17:45.867679    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5\": container with ID starting with 66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5 not found: ID does not exist" containerID="66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5"
Oct 30 22:17:45.867773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.867706    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5"} err="failed to get container status \"66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5\": rpc error: code = NotFound desc = could not find container \"66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5\": container with ID starting with 66bc58db944973c0438247a4de017b430aed651661c20680a44d3ca508c2bca5 not found: ID does not exist"
Oct 30 22:17:45.867773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.867722    5503 scope.go:122] "RemoveContainer" containerID="e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265"
Oct 30 22:17:45.876348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.876319    5503 scope.go:122] "RemoveContainer" containerID="e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265"
Oct 30 22:17:45.876621 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:17:45.876600    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265\": container with ID starting with e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265 not found: ID does not exist" containerID="e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265"
Oct 30 22:17:45.876715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.876625    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265"} err="failed to get container status \"e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265\": rpc error: code = NotFound desc = could not find container \"e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265\": container with ID starting with e9df59cb57a5cc973e850427d5a773cc59de72b2f843f1f59fdfac1272e88265 not found: ID does not exist"
Oct 30 22:17:45.876715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.876655    5503 scope.go:122] "RemoveContainer" containerID="eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f"
Oct 30 22:17:45.885786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.885759    5503 scope.go:122] "RemoveContainer" containerID="eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f"
Oct 30 22:17:45.886065 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:17:45.886045    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f\": container with ID starting with eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f not found: ID does not exist" containerID="eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f"
Oct 30 22:17:45.886113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:45.886077    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f"} err="failed to get container status \"eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f\": rpc error: code = NotFound desc = could not find container \"eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f\": container with ID starting with eee0167ee43337bc12f2b48d1a5e125e51efe5020998c4b16ce3b262b5d3bd6f not found: ID does not exist"
Oct 30 22:17:46.023350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:46.023318    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"]
Oct 30 22:17:46.074009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:46.073925    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-89bbp"]
Oct 30 22:17:46.139831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:46.139778    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"]
Oct 30 22:17:46.146591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:46.146566    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-8zfgj"]
Oct 30 22:17:46.259049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:46.259016    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"]
Oct 30 22:17:46.263170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:46.263140    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4cf21208-openshift-default-6b4f6cc45c-cft8t"]
Oct 30 22:17:46.664116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:46.664083    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:47.797484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:47.797447    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24807402-0466-48b6-b00f-010db3b9dd67" path="/var/lib/kubelet/pods/24807402-0466-48b6-b00f-010db3b9dd67/volumes"
Oct 30 22:17:47.798063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:47.798047    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" path="/var/lib/kubelet/pods/3ac5603e-96d3-4cf8-8123-e896135b277f/volumes"
Oct 30 22:17:47.798469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:47.798454    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" path="/var/lib/kubelet/pods/be720949-f679-4113-bb9e-3bc80e3c4cf1/volumes"
Oct 30 22:17:49.206198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206168    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"]
Oct 30 22:17:49.206699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206681    5503 cpu_manager.go:414] "containerMap: removing container" podUID="6ef1151b-5089-4c97-b231-618362103767" containerName="revoke-keys"
Oct 30 22:17:49.206752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206700    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef1151b-5089-4c97-b231-618362103767" containerName="revoke-keys"
Oct 30 22:17:49.206752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206723    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy"
Oct 30 22:17:49.206752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206728    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy"
Oct 30 22:17:49.206752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206739    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy"
Oct 30 22:17:49.206752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206745    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206760    5503 cpu_manager.go:414] "containerMap: removing container" podUID="8fa83512-318e-4a91-82cd-25a1404dabcd" containerName="main"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206768    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa83512-318e-4a91-82cd-25a1404dabcd" containerName="main"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206784    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5934ca8e-788f-42d1-9384-646deb04c0ef" containerName="revoke-keys"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206789    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5934ca8e-788f-42d1-9384-646deb04c0ef" containerName="revoke-keys"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206800    5503 cpu_manager.go:414] "containerMap: removing container" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206804    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206816    5503 cpu_manager.go:414] "containerMap: removing container" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy"
Oct 30 22:17:49.206908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206821    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy"
Oct 30 22:17:49.207314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206924    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef1151b-5089-4c97-b231-618362103767" containerName="revoke-keys"
Oct 30 22:17:49.207314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206935    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy"
Oct 30 22:17:49.207314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206942    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac5603e-96d3-4cf8-8123-e896135b277f" containerName="istio-proxy"
Oct 30 22:17:49.207314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206949    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa83512-318e-4a91-82cd-25a1404dabcd" containerName="main"
Oct 30 22:17:49.207314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206956    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5934ca8e-788f-42d1-9384-646deb04c0ef" containerName="revoke-keys"
Oct 30 22:17:49.207314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206963    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="be720949-f679-4113-bb9e-3bc80e3c4cf1" containerName="istio-proxy"
Oct 30 22:17:49.207314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.206969    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="24807402-0466-48b6-b00f-010db3b9dd67" containerName="istio-proxy"
Oct 30 22:17:49.212232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.212213    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.260921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.260888    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.261115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.261051    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htlfh\" (UniqueName: \"kubernetes.io/projected/ce98b14f-e1b5-4885-b656-2262f87c17f5-kube-api-access-htlfh\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.261268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.261244    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.264272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.264247    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"]
Oct 30 22:17:49.312724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.312691    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:17:49.312724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.312714    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:17:49.312898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.312696    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:49.312898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.312757    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:17:49.362343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.362316    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htlfh\" (UniqueName: \"kubernetes.io/projected/ce98b14f-e1b5-4885-b656-2262f87c17f5-kube-api-access-htlfh\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.362484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.362431    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.362484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.362479    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.363214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.363193    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.364849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.364831    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.424228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.424198    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htlfh\" (UniqueName: \"kubernetes.io/projected/ce98b14f-e1b5-4885-b656-2262f87c17f5-kube-api-access-htlfh\") pod \"maas-api-revoke-sub-b0e30260-m2gnx\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.525373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.525302    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:49.929990 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:17:49.929957    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce98b14f_e1b5_4885_b656_2262f87c17f5.slice/crio-02ee3783dc130ebc325c83cd01efad4ebfb72fca168fde09e86d90ebbd581574 WatchSource:0}: Error finding container 02ee3783dc130ebc325c83cd01efad4ebfb72fca168fde09e86d90ebbd581574: Status 404 returned error can't find the container with id 02ee3783dc130ebc325c83cd01efad4ebfb72fca168fde09e86d90ebbd581574
Oct 30 22:17:49.933001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:49.932974    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"]
Oct 30 22:17:50.704601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:50.704512    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:50.914524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:50.914489    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx" event={"ID":"ce98b14f-e1b5-4885-b656-2262f87c17f5","Type":"ContainerStarted","Data":"ce98d68d2da2c793f13f0faafbdccfafb9259c1976fc1bc439bf59ec87ea5ef4"}
Oct 30 22:17:50.914835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:50.914816    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx" event={"ID":"ce98b14f-e1b5-4885-b656-2262f87c17f5","Type":"ContainerStarted","Data":"02ee3783dc130ebc325c83cd01efad4ebfb72fca168fde09e86d90ebbd581574"}
Oct 30 22:17:51.033463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:51.033403    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx" podStartSLOduration=2.0333889 podStartE2EDuration="2.0333889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:51.031607737 +0000 UTC m=+1847.366184506" watchObservedRunningTime="2026-10-30 22:17:51.0333889 +0000 UTC m=+1847.367965731"
Oct 30 22:17:51.110661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:51.103527    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"]
Oct 30 22:17:51.110661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:51.107039    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-322f22e6-6w47j"]
Oct 30 22:17:51.789092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:51.789058    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8113f99c-1f5b-4006-bb0b-893756ac01f4" path="/var/lib/kubelet/pods/8113f99c-1f5b-4006-bb0b-893756ac01f4/volumes"
Oct 30 22:17:51.922320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:51.922282    5503 generic.go:356] "Generic (PLEG): container finished" podID="ce98b14f-e1b5-4885-b656-2262f87c17f5" containerID="ce98d68d2da2c793f13f0faafbdccfafb9259c1976fc1bc439bf59ec87ea5ef4" exitCode=0
Oct 30 22:17:51.922320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:51.922323    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx" event={"ID":"ce98b14f-e1b5-4885-b656-2262f87c17f5","Type":"ContainerDied","Data":"ce98d68d2da2c793f13f0faafbdccfafb9259c1976fc1bc439bf59ec87ea5ef4"}
Oct 30 22:17:52.615505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:52.615475    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:52.895183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:52.895096    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.078698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.078671    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:53.138425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.138396    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce98b14f-e1b5-4885-b656-2262f87c17f5-kube-api-access-htlfh\" (UniqueName: \"kubernetes.io/projected/ce98b14f-e1b5-4885-b656-2262f87c17f5-kube-api-access-htlfh\") pod \"ce98b14f-e1b5-4885-b656-2262f87c17f5\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") "
Oct 30 22:17:53.138618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.138504    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-openshift-ca\") pod \"ce98b14f-e1b5-4885-b656-2262f87c17f5\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") "
Oct 30 22:17:53.138618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.138581    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-xks-ca\") pod \"ce98b14f-e1b5-4885-b656-2262f87c17f5\" (UID: \"ce98b14f-e1b5-4885-b656-2262f87c17f5\") "
Oct 30 22:17:53.138897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.138864    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-openshift-ca" pod "ce98b14f-e1b5-4885-b656-2262f87c17f5" (UID: "ce98b14f-e1b5-4885-b656-2262f87c17f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:53.139048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.139031    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:53.140551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.140533    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce98b14f-e1b5-4885-b656-2262f87c17f5-kube-api-access-htlfh" pod "ce98b14f-e1b5-4885-b656-2262f87c17f5" (UID: "ce98b14f-e1b5-4885-b656-2262f87c17f5"). InnerVolumeSpecName "kube-api-access-htlfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:53.140648 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.140612    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-xks-ca" pod "ce98b14f-e1b5-4885-b656-2262f87c17f5" (UID: "ce98b14f-e1b5-4885-b656-2262f87c17f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:53.165113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.165079    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.241135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.241101    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce98b14f-e1b5-4885-b656-2262f87c17f5-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:53.241393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.241377    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htlfh\" (UniqueName: \"kubernetes.io/projected/ce98b14f-e1b5-4885-b656-2262f87c17f5-kube-api-access-htlfh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:53.364062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.364032    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"]
Oct 30 22:17:53.367211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.367188    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-427d70-wzjr9"]
Oct 30 22:17:53.789212 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.789178    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5934ca8e-788f-42d1-9384-646deb04c0ef" path="/var/lib/kubelet/pods/5934ca8e-788f-42d1-9384-646deb04c0ef/volumes"
Oct 30 22:17:53.804566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.804539    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:53.935624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.935581    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx" event={"ID":"ce98b14f-e1b5-4885-b656-2262f87c17f5","Type":"ContainerDied","Data":"02ee3783dc130ebc325c83cd01efad4ebfb72fca168fde09e86d90ebbd581574"}
Oct 30 22:17:53.936021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.935604    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"
Oct 30 22:17:53.936021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:53.935663    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ee3783dc130ebc325c83cd01efad4ebfb72fca168fde09e86d90ebbd581574"
Oct 30 22:17:55.589487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:55.589437    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:55.741939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:55.741900    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:55.891372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:55.891293    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:57.134846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:57.133852    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"]
Oct 30 22:17:57.134846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:57.134103    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" containerID="cri-o://b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0" gracePeriod=30
Oct 30 22:17:57.174669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:57.174616    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"]
Oct 30 22:17:57.175181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:57.175138    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" containerID="cri-o://7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc" gracePeriod=30
Oct 30 22:17:57.456902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:57.456860    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:17:58.101104 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:17:58.101064    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:17:58.820725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:58.820685    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:58.821119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:58.820746    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:59.935746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:17:59.935711    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.361987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:00.361901    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"]
Oct 30 22:18:00.362159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:00.362132    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" containerID="cri-o://4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061" gracePeriod=30
Oct 30 22:18:00.404605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:00.404568    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.563200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:00.563171    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:00.750847 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:00.750808    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:01.786088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:01.786057    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.374950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.374879    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.555664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.548423    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.701878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.701804    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:18:02.711695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.711491    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:02.711971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.711922    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:02.729892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.729866    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:18:02.795758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.795720    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-data\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.795786    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-kube-api-access-vk9lp\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-kube-api-access-vk9lp\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.795844    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-podinfo\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.795877    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/289cdaae-6028-42e7-a615-c4358d99287e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289cdaae-6028-42e7-a615-c4358d99287e-istio-podinfo\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.795913    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-token\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-token\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.795960    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-credential-socket\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796021    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-certs\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796059    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-istio-token\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-istio-token\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796092    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-certs\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796159    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289cdaae-6028-42e7-a615-c4358d99287e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289cdaae-6028-42e7-a615-c4358d99287e-istiod-ca-cert\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796230    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-socket\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796280    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-socket\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796051    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-data" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796307    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-kube-api-access-lxhgs\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-kube-api-access-lxhgs\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796383    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istiod-ca-cert\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796407    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-data\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796443    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-envoy\") pod \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\" (UID: \"5845e6a9-82a9-4bee-86bf-ca8637bf48b7\") "
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796464    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-credential-socket\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.796858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.796500    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-envoy\") pod \"289cdaae-6028-42e7-a615-c4358d99287e\" (UID: \"289cdaae-6028-42e7-a615-c4358d99287e\") "
Oct 30 22:18:02.797305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.797130    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.798165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.797540    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-certs" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.798165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.797678    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-credential-socket" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.798165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.797695    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-socket" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.798165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.797839    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-certs" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.798165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.797958    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-socket" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.798500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.798279    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-credential-socket" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.798500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.798452    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289cdaae-6028-42e7-a615-c4358d99287e-istiod-ca-cert" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:02.798889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.798802    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istiod-ca-cert" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:02.798889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.798867    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-data" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.799496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.799470    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-kube-api-access-lxhgs" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "kube-api-access-lxhgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:02.799628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.799544    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-envoy" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.801226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.801201    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-istio-token" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:02.801377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.801306    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:02.801465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.801379    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-token" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:02.801797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.801774    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-podinfo" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:02.801987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.801962    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-kube-api-access-vk9lp" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "kube-api-access-vk9lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:02.802586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.802561    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-envoy" pod "5845e6a9-82a9-4bee-86bf-ca8637bf48b7" (UID: "5845e6a9-82a9-4bee-86bf-ca8637bf48b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:02.802819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.802799    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289cdaae-6028-42e7-a615-c4358d99287e-istio-podinfo" pod "289cdaae-6028-42e7-a615-c4358d99287e" (UID: "289cdaae-6028-42e7-a615-c4358d99287e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:02.898707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898667    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.898707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898698    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.898707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898709    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.898707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898717    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898726    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk9lp\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-kube-api-access-vk9lp\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898736    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898744    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289cdaae-6028-42e7-a615-c4358d99287e-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898753    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898764    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898772    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898780    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289cdaae-6028-42e7-a615-c4358d99287e-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898789    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898797    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289cdaae-6028-42e7-a615-c4358d99287e-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898804    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898812    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289cdaae-6028-42e7-a615-c4358d99287e-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898820    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxhgs\" (UniqueName: \"kubernetes.io/projected/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-kube-api-access-lxhgs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:02.899071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:02.898828    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5845e6a9-82a9-4bee-86bf-ca8637bf48b7-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:03.035387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.035289    5503 generic.go:356] "Generic (PLEG): container finished" podID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerID="b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0" exitCode=0
Oct 30 22:18:03.035387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.035357    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"
Oct 30 22:18:03.035620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.035387    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" event={"ID":"5845e6a9-82a9-4bee-86bf-ca8637bf48b7","Type":"ContainerDied","Data":"b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0"}
Oct 30 22:18:03.035620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.035448    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" event={"ID":"5845e6a9-82a9-4bee-86bf-ca8637bf48b7","Type":"ContainerDied","Data":"a738e15b1bcfb52c8f08e65c58b99ba8026c05ea7646491a1e9c3376c88aa7f9"}
Oct 30 22:18:03.035620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.035480    5503 scope.go:122] "RemoveContainer" containerID="b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0"
Oct 30 22:18:03.038919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.038893    5503 generic.go:356] "Generic (PLEG): container finished" podID="289cdaae-6028-42e7-a615-c4358d99287e" containerID="7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc" exitCode=0
Oct 30 22:18:03.039082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.039063    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" event={"ID":"289cdaae-6028-42e7-a615-c4358d99287e","Type":"ContainerDied","Data":"7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc"}
Oct 30 22:18:03.039201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.039109    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn" event={"ID":"289cdaae-6028-42e7-a615-c4358d99287e","Type":"ContainerDied","Data":"9177bcc665a8d8a2279f899ae92de161b617d3e76fb0310ba7bddef0b0ef3c2f"}
Oct 30 22:18:03.039201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.039180    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"
Oct 30 22:18:03.050328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.050310    5503 scope.go:122] "RemoveContainer" containerID="b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0"
Oct 30 22:18:03.050813 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:18:03.050790    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0\": container with ID starting with b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0 not found: ID does not exist" containerID="b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0"
Oct 30 22:18:03.050983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.050958    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0"} err="failed to get container status \"b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0\": rpc error: code = NotFound desc = could not find container \"b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0\": container with ID starting with b5e920e3afb3fa788df8895d87a273851d48840415080f65e117108ecb4af3e0 not found: ID does not exist"
Oct 30 22:18:03.051078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.051065    5503 scope.go:122] "RemoveContainer" containerID="7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc"
Oct 30 22:18:03.061760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.061729    5503 scope.go:122] "RemoveContainer" containerID="7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc"
Oct 30 22:18:03.062049 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:18:03.062022    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc\": container with ID starting with 7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc not found: ID does not exist" containerID="7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc"
Oct 30 22:18:03.062120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.062058    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc"} err="failed to get container status \"7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc\": rpc error: code = NotFound desc = could not find container \"7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc\": container with ID starting with 7cb0dddf68c008adb320025e5add9ef7481b060483b63653dba91a3eea6eaedc not found: ID does not exist"
Oct 30 22:18:03.206113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.206080    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"]
Oct 30 22:18:03.210695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.210666    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-8dgjn"]
Oct 30 22:18:03.324724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.324615    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"]
Oct 30 22:18:03.328801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.328769    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs"]
Oct 30 22:18:03.373355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.373317    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:18:03.373527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.373402    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-wvgzs" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:03.792564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.792524    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289cdaae-6028-42e7-a615-c4358d99287e" path="/var/lib/kubelet/pods/289cdaae-6028-42e7-a615-c4358d99287e/volumes"
Oct 30 22:18:03.793246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:03.793225    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" path="/var/lib/kubelet/pods/5845e6a9-82a9-4bee-86bf-ca8637bf48b7/volumes"
Oct 30 22:18:04.502654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:04.502599    5503 scope.go:122] "RemoveContainer" containerID="937db330c52004036f59f493e3f08be032309bbb12d9255d698ff5a0f92f21e0"
Oct 30 22:18:05.626566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.626505    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:18:05.734918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.734890    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-credential-socket\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.734931    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-certs\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.734960    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-kube-api-access-t2tww\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-kube-api-access-t2tww\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.734976    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-envoy\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735001    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13c34e5e-9c39-46b2-808f-83192790e90a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c34e5e-9c39-46b2-808f-83192790e90a-istio-podinfo\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735025    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-istio-token\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-istio-token\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735132    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c34e5e-9c39-46b2-808f-83192790e90a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c34e5e-9c39-46b2-808f-83192790e90a-istiod-ca-cert\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735166    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-data\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735197    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-certs" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:05.735380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735248    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-credential-socket" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:05.735573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735344    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-socket\") pod \"13c34e5e-9c39-46b2-808f-83192790e90a\" (UID: \"13c34e5e-9c39-46b2-808f-83192790e90a\") "
Oct 30 22:18:05.735573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735469    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-data" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:05.735573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735509    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c34e5e-9c39-46b2-808f-83192790e90a-istiod-ca-cert" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:05.735804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735779    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-socket" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:05.735959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735940    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c34e5e-9c39-46b2-808f-83192790e90a-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.736010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735963    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.736010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735979    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.736010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.735994    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.736122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.736009    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.737359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.737337    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-istio-token" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:05.737469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.737375    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-envoy" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:05.737724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.737689    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13c34e5e-9c39-46b2-808f-83192790e90a-istio-podinfo" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:05.737724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.737717    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-kube-api-access-t2tww" pod "13c34e5e-9c39-46b2-808f-83192790e90a" (UID: "13c34e5e-9c39-46b2-808f-83192790e90a"). InnerVolumeSpecName "kube-api-access-t2tww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:05.837587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.837554    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tww\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-kube-api-access-t2tww\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.837587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.837581    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c34e5e-9c39-46b2-808f-83192790e90a-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.837587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.837591    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c34e5e-9c39-46b2-808f-83192790e90a-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:05.837804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:05.837600    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c34e5e-9c39-46b2-808f-83192790e90a-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:06.062040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.062003    5503 generic.go:356] "Generic (PLEG): container finished" podID="13c34e5e-9c39-46b2-808f-83192790e90a" containerID="4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061" exitCode=0
Oct 30 22:18:06.062225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.062057    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"
Oct 30 22:18:06.062225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.062079    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" event={"ID":"13c34e5e-9c39-46b2-808f-83192790e90a","Type":"ContainerDied","Data":"4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061"}
Oct 30 22:18:06.062225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.062132    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck" event={"ID":"13c34e5e-9c39-46b2-808f-83192790e90a","Type":"ContainerDied","Data":"4c4ecead0676e865be5be7d2582daaf13ee10943d2ced5781fe0e69471077b7d"}
Oct 30 22:18:06.062225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.062153    5503 scope.go:122] "RemoveContainer" containerID="4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061"
Oct 30 22:18:06.071941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.071926    5503 scope.go:122] "RemoveContainer" containerID="4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061"
Oct 30 22:18:06.072217 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:18:06.072197    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061\": container with ID starting with 4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061 not found: ID does not exist" containerID="4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061"
Oct 30 22:18:06.072252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.072227    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061"} err="failed to get container status \"4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061\": rpc error: code = NotFound desc = could not find container \"4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061\": container with ID starting with 4caead9a98d510457e71b8e840a39f8b204a379001f4c5a1aea26de085f72061 not found: ID does not exist"
Oct 30 22:18:06.236677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.236625    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"]
Oct 30 22:18:06.273084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.273051    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:06.287479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:06.287450    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-rbfck"]
Oct 30 22:18:07.128661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:07.128597    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:07.609444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:07.609405    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:07.767823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:07.767781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:07.789603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:07.789572    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" path="/var/lib/kubelet/pods/13c34e5e-9c39-46b2-808f-83192790e90a/volumes"
Oct 30 22:18:07.930245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:07.930206    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:10.847809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:10.847772    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:11.042771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.042736    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:11.261648 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.261603    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"]
Oct 30 22:18:11.262062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262048    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy"
Oct 30 22:18:11.262121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262063    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy"
Oct 30 22:18:11.262121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262086    5503 cpu_manager.go:414] "containerMap: removing container" podUID="ce98b14f-e1b5-4885-b656-2262f87c17f5" containerName="revoke-keys"
Oct 30 22:18:11.262121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262091    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce98b14f-e1b5-4885-b656-2262f87c17f5" containerName="revoke-keys"
Oct 30 22:18:11.262121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262100    5503 cpu_manager.go:414] "containerMap: removing container" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy"
Oct 30 22:18:11.262121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262105    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262124    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262129    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262141    5503 cpu_manager.go:414] "containerMap: removing container" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262146    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262218    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5845e6a9-82a9-4bee-86bf-ca8637bf48b7" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262228    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce98b14f-e1b5-4885-b656-2262f87c17f5" containerName="revoke-keys"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262236    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c34e5e-9c39-46b2-808f-83192790e90a" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262244    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b132c2-4393-49be-9adb-dd50b45d2322" containerName="istio-proxy"
Oct 30 22:18:11.262273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.262251    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="289cdaae-6028-42e7-a615-c4358d99287e" containerName="istio-proxy"
Oct 30 22:18:11.266364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.266346    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.316198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.316163    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vdzd\" (UniqueName: \"kubernetes.io/projected/970a9597-129f-4a5e-a976-7092f7e07590-kube-api-access-9vdzd\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.316438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.316403    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970a9597-129f-4a5e-a976-7092f7e07590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.322000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.321969    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"]
Oct 30 22:18:11.379188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.377022    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0e90fe-dockercfg-lm97r\""
Oct 30 22:18:11.380133 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.380105    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0e90fe\""
Oct 30 22:18:11.423972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.420151    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970a9597-129f-4a5e-a976-7092f7e07590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.423972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.420244    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vdzd\" (UniqueName: \"kubernetes.io/projected/970a9597-129f-4a5e-a976-7092f7e07590-kube-api-access-9vdzd\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.423972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.421825    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970a9597-129f-4a5e-a976-7092f7e07590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.444465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.440430    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"]
Oct 30 22:18:11.452967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.450109    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.479951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.479921    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"]
Oct 30 22:18:11.484598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.484571    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vdzd\" (UniqueName: \"kubernetes.io/projected/970a9597-129f-4a5e-a976-7092f7e07590-kube-api-access-9vdzd\") pod \"payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.521577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.521501    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed201742-12c7-4fde-8c56-705a8de721ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.521729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.521625    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls47k\" (UniqueName: \"kubernetes.io/projected/ed201742-12c7-4fde-8c56-705a8de721ca-kube-api-access-ls47k\") pod \"payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.579054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.579012    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:11.623304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.623272    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed201742-12c7-4fde-8c56-705a8de721ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.623444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.623331    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls47k\" (UniqueName: \"kubernetes.io/projected/ed201742-12c7-4fde-8c56-705a8de721ca-kube-api-access-ls47k\") pod \"payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.624206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.624180    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed201742-12c7-4fde-8c56-705a8de721ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.685628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.685592    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls47k\" (UniqueName: \"kubernetes.io/projected/ed201742-12c7-4fde-8c56-705a8de721ca-kube-api-access-ls47k\") pod \"payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.799613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.799541    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:11.954553 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:18:11.954501    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970a9597_129f_4a5e_a976_7092f7e07590.slice/crio-da63d2c1151ade42910b3d6f86c1cf4d0be6fe9ddd41410c94f64600d28c145e WatchSource:0}: Error finding container da63d2c1151ade42910b3d6f86c1cf4d0be6fe9ddd41410c94f64600d28c145e: Status 404 returned error can't find the container with id da63d2c1151ade42910b3d6f86c1cf4d0be6fe9ddd41410c94f64600d28c145e
Oct 30 22:18:11.955806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:11.955779    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"]
Oct 30 22:18:12.023178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.023153    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:12.121255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.121174    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t" event={"ID":"970a9597-129f-4a5e-a976-7092f7e07590","Type":"ContainerStarted","Data":"edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474"}
Oct 30 22:18:12.121255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.121221    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t" event={"ID":"970a9597-129f-4a5e-a976-7092f7e07590","Type":"ContainerStarted","Data":"da63d2c1151ade42910b3d6f86c1cf4d0be6fe9ddd41410c94f64600d28c145e"}
Oct 30 22:18:12.121492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.121394    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:12.174679 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:18:12.174645    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded201742_12c7_4fde_8c56_705a8de721ca.slice/crio-d191d8deae94d8a8f0626aac08f67a2ea4604e19b7df82df051433fd9c81ca2e WatchSource:0}: Error finding container d191d8deae94d8a8f0626aac08f67a2ea4604e19b7df82df051433fd9c81ca2e: Status 404 returned error can't find the container with id d191d8deae94d8a8f0626aac08f67a2ea4604e19b7df82df051433fd9c81ca2e
Oct 30 22:18:12.176896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.176869    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"]
Oct 30 22:18:12.180668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.180646    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:12.243656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.240956    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t" podStartSLOduration=1.240936335 podStartE2EDuration="1.240936335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:12.240596986 +0000 UTC m=+1868.575173758" watchObservedRunningTime="2026-10-30 22:18:12.240936335 +0000 UTC m=+1868.575513105"
Oct 30 22:18:12.338247 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:12.338215    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:13.128599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:13.128563    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx" event={"ID":"ed201742-12c7-4fde-8c56-705a8de721ca","Type":"ContainerStarted","Data":"023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847"}
Oct 30 22:18:13.129025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:13.128625    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx" event={"ID":"ed201742-12c7-4fde-8c56-705a8de721ca","Type":"ContainerStarted","Data":"d191d8deae94d8a8f0626aac08f67a2ea4604e19b7df82df051433fd9c81ca2e"}
Oct 30 22:18:13.129025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:13.128676    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:13.249419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:13.249363    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx" podStartSLOduration=2.249349904 podStartE2EDuration="2.249349904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:13.247310425 +0000 UTC m=+1869.581887194" watchObservedRunningTime="2026-10-30 22:18:13.249349904 +0000 UTC m=+1869.583926675"
Oct 30 22:18:16.852476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:16.852444    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:17.257000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:17.256967    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:18:19.856252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:19.856211    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"]
Oct 30 22:18:19.858454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:19.857179    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" containerID="cri-o://2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986" gracePeriod=30
Oct 30 22:18:20.308802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:20.308765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:21.411453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:21.411417    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:22.058269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:22.058229    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"]
Oct 30 22:18:22.108167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:22.108130    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107aa622-lhqjq"]
Oct 30 22:18:23.131863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:23.131835    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:18:23.574309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:23.574276    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:23.729138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:23.729104    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:23.789288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:23.789254    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5062c257-b1ac-42b1-8a9f-9d3fd69bfd86" path="/var/lib/kubelet/pods/5062c257-b1ac-42b1-8a9f-9d3fd69bfd86/volumes"
Oct 30 22:18:24.138403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:24.138368    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:18:25.303909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:25.303874    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-zjzp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:18:25.304308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:25.303927    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:18:26.057078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.057050    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:18:26.175094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175060    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-certs\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175123    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-credential-socket\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175216    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-token\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-token\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175260    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istiod-ca-cert\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175293    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-kube-api-access-qv8n7\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-kube-api-access-qv8n7\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175348    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-certs" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:26.175464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175369    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-socket\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175408    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-envoy\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175452    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-credential-socket" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:26.175702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175474    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-podinfo\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175614    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-data\") pod \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\" (UID: \"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7\") "
Oct 30 22:18:26.175885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175707    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-socket" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:26.175885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175742    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istiod-ca-cert" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:26.176008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.175952    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-data" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:26.176168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.176134    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.176168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.176161    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.176309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.176178    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.176309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.176194    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.176309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.176208    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.177870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.177718    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-kube-api-access-qv8n7" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "kube-api-access-qv8n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:26.177988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.177877    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-podinfo" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:26.177988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.177908    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-token" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:26.178138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.178115    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-envoy" pod "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" (UID: "cdd4432e-5f73-4ef7-9e6f-2a947b0766e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:26.241531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.241493    5503 generic.go:356] "Generic (PLEG): container finished" podID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerID="2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986" exitCode=0
Oct 30 22:18:26.241780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.241564    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"
Oct 30 22:18:26.241780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.241693    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" event={"ID":"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7","Type":"ContainerDied","Data":"2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986"}
Oct 30 22:18:26.241780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.241740    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6" event={"ID":"cdd4432e-5f73-4ef7-9e6f-2a947b0766e7","Type":"ContainerDied","Data":"ebf9b3a0d693821400cb90dd85eb87027188dd746a69c42d2c2ecd7276615df5"}
Oct 30 22:18:26.241780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.241768    5503 scope.go:122] "RemoveContainer" containerID="2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986"
Oct 30 22:18:26.255340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.255319    5503 scope.go:122] "RemoveContainer" containerID="2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986"
Oct 30 22:18:26.255607 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:18:26.255587    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986\": container with ID starting with 2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986 not found: ID does not exist" containerID="2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986"
Oct 30 22:18:26.255699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.255616    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986"} err="failed to get container status \"2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986\": rpc error: code = NotFound desc = could not find container \"2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986\": container with ID starting with 2b91cbecc5e9c6d471a917ae50a1a22de4e8bed326ef7af38716972dfc312986 not found: ID does not exist"
Oct 30 22:18:26.278013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.277983    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.278134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.278018    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv8n7\" (UniqueName: \"kubernetes.io/projected/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-kube-api-access-qv8n7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.278134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.278035    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.278134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.278050    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:26.413859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.413822    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"]
Oct 30 22:18:26.417799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:26.417775    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-zjzp6"]
Oct 30 22:18:27.789091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:27.789049    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" path="/var/lib/kubelet/pods/cdd4432e-5f73-4ef7-9e6f-2a947b0766e7/volumes"
Oct 30 22:18:28.098574 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:18:28.098527    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:18:29.628416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:29.628379    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:34.217505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:34.217467    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:35.003483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.003444    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"]
Oct 30 22:18:35.003969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.003955    5503 cpu_manager.go:414] "containerMap: removing container" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy"
Oct 30 22:18:35.004024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.003970    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy"
Oct 30 22:18:35.004063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.004057    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd4432e-5f73-4ef7-9e6f-2a947b0766e7" containerName="istio-proxy"
Oct 30 22:18:35.008778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.008760    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073443    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073492    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073523    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073615    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073685    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073709    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbh55\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-kube-api-access-qbh55\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073738    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b58ca3-706f-4c30-9608-d0cbf626e912-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073783    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.073940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.073860    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.127063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.127026    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"]
Oct 30 22:18:35.174688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.174631    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.174873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.174818    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.174873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.174855    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.174900    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.174931    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.174960    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbh55\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-kube-api-access-qbh55\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175166 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.175005    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b58ca3-706f-4c30-9608-d0cbf626e912-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175166 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.175037    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175166 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.175131    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.175196    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.175457    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-data\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.175580    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-certs\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.175979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.175956    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b58ca3-706f-4c30-9608-d0cbf626e912-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.177008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.176978    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-credential-socket\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.177857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.177831    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.178058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.178031    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-envoy\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.236884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.236855    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-token\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.238082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.238054    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbh55\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-kube-api-access-qbh55\") pod \"maas-default-gateway-openshift-default-6f54779779-svxzp\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.322334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.322269    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:35.749718 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:18:35.749688    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b58ca3_706f_4c30_9608_d0cbf626e912.slice/crio-56be2aa23a6b2957605f6a2531c1c91f07e7a330136a25f6d9c01cbb6124e258 WatchSource:0}: Error finding container 56be2aa23a6b2957605f6a2531c1c91f07e7a330136a25f6d9c01cbb6124e258: Status 404 returned error can't find the container with id 56be2aa23a6b2957605f6a2531c1c91f07e7a330136a25f6d9c01cbb6124e258
Oct 30 22:18:35.750694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.750666    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"]
Oct 30 22:18:35.752036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.751999    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:18:35.752143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.752064    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:18:35.752143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:35.752093    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:18:36.316195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:36.316153    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" event={"ID":"79b58ca3-706f-4c30-9608-d0cbf626e912","Type":"ContainerStarted","Data":"3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64"}
Oct 30 22:18:36.316834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:36.316799    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" event={"ID":"79b58ca3-706f-4c30-9608-d0cbf626e912","Type":"ContainerStarted","Data":"56be2aa23a6b2957605f6a2531c1c91f07e7a330136a25f6d9c01cbb6124e258"}
Oct 30 22:18:36.326003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:36.325972    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:36.335825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:36.335787    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:36.336132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:36.336096    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:36.442363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:36.442300    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podStartSLOduration=2.4422794420000002 podStartE2EDuration="2.442279442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:36.441010348 +0000 UTC m=+1892.775587116" watchObservedRunningTime="2026-10-30 22:18:36.442279442 +0000 UTC m=+1892.776856214"
Oct 30 22:18:36.474668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:36.474619    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:37.326007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:37.325973    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:37.326802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:37.326770    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:38.339804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.339767    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:38.340255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.339831    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:38.411165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.411134    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"]
Oct 30 22:18:38.425662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.424918    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.471442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.471400    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"]
Oct 30 22:18:38.503516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.503485    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:38.503671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.503515    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:18:38.539903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.539866    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:18:38.540079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.539979    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:18:38.540809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.540784    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.540935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.540831    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.541022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.540997    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmfx\" (UniqueName: \"kubernetes.io/projected/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-kube-api-access-8vmfx\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.642419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.642334    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.642419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.642373    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.642621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.642437    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmfx\" (UniqueName: \"kubernetes.io/projected/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-kube-api-access-8vmfx\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.643173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.643141    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.644863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.644844    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.704252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.704224    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmfx\" (UniqueName: \"kubernetes.io/projected/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-kube-api-access-8vmfx\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:38.751950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:38.751909    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:39.002799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:39.002760    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:39.143331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:39.143302    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"]
Oct 30 22:18:39.147939 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:18:39.147911    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad43b70d_bcb0_4e9e_8e33_248d53f4ee94.slice/crio-65d8acf145c3b9bf7ea9cb9b38fa70ff0b89792d9d411f2d1d8e4d2eb9dee3ec WatchSource:0}: Error finding container 65d8acf145c3b9bf7ea9cb9b38fa70ff0b89792d9d411f2d1d8e4d2eb9dee3ec: Status 404 returned error can't find the container with id 65d8acf145c3b9bf7ea9cb9b38fa70ff0b89792d9d411f2d1d8e4d2eb9dee3ec
Oct 30 22:18:39.202658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:39.195204    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:39.339727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:39.334117    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:39.339727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:39.334174    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:39.351497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:39.351472    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg" event={"ID":"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94","Type":"ContainerStarted","Data":"65d8acf145c3b9bf7ea9cb9b38fa70ff0b89792d9d411f2d1d8e4d2eb9dee3ec"}
Oct 30 22:18:40.324430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:40.323985    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:40.324430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:40.324050    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:40.398692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:40.398652    5503 generic.go:356] "Generic (PLEG): container finished" podID="ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" containerID="9e732728939ba0df28ce883f2ed6291d6727e9474356c7175ecc24028d101666" exitCode=0
Oct 30 22:18:40.399089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:40.398743    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg" event={"ID":"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94","Type":"ContainerDied","Data":"9e732728939ba0df28ce883f2ed6291d6727e9474356c7175ecc24028d101666"}
Oct 30 22:18:40.458143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:40.458109    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:41.333660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.328002    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:41.333660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.328064    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:41.604665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.604621    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:41.609071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.609050    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-xks-ca\") pod \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") "
Oct 30 22:18:41.609166 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.609097    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-openshift-ca\") pod \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") "
Oct 30 22:18:41.609251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.609229    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-kube-api-access-8vmfx\" (UniqueName: \"kubernetes.io/projected/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-kube-api-access-8vmfx\") pod \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\" (UID: \"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94\") "
Oct 30 22:18:41.609514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.609492    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-openshift-ca" pod "ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" (UID: "ad43b70d-bcb0-4e9e-8e33-248d53f4ee94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:41.609933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.609910    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.611390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.611366    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-kube-api-access-8vmfx" pod "ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" (UID: "ad43b70d-bcb0-4e9e-8e33-248d53f4ee94"). InnerVolumeSpecName "kube-api-access-8vmfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.611571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.611547    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-xks-ca" pod "ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" (UID: "ad43b70d-bcb0-4e9e-8e33-248d53f4ee94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:41.711413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.711378    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmfx\" (UniqueName: \"kubernetes.io/projected/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-kube-api-access-8vmfx\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.711413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:41.711406    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.323542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:42.323506    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:42.323724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:42.323561    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:42.412504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:42.412475    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"
Oct 30 22:18:42.412737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:42.412473    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg" event={"ID":"ad43b70d-bcb0-4e9e-8e33-248d53f4ee94","Type":"ContainerDied","Data":"65d8acf145c3b9bf7ea9cb9b38fa70ff0b89792d9d411f2d1d8e4d2eb9dee3ec"}
Oct 30 22:18:42.412737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:42.412595    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d8acf145c3b9bf7ea9cb9b38fa70ff0b89792d9d411f2d1d8e4d2eb9dee3ec"
Oct 30 22:18:42.495491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:42.495451    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"]
Oct 30 22:18:42.495718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:42.495678    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy" containerID="cri-o://97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e" gracePeriod=30
Oct 30 22:18:43.322827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:43.322784    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 30 22:18:43.323337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:43.322841    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 30 22:18:44.327488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:44.327448    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:44.328037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:44.328016    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:44.328912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:44.328889    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:18:46.212761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:46.212718    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:47.752207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.752179    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:18:47.782536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782462    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istiod-ca-cert\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782522    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-data\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782571    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-token\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-token\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782656    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-kube-api-access-wkx54\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-kube-api-access-wkx54\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782713    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-credential-socket\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782741    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-certs\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782790    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-socket\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782853    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-podinfo\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782885    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-envoy\") pod \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\" (UID: \"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5\") "
Oct 30 22:18:47.782988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782896    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-data" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.782988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.782945    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istiod-ca-cert" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:47.783286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.783098    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-socket" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.783286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.783194    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-certs" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.783524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.783500    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-credential-socket" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.783623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.783537    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.783623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.783556    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.783623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.783570    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.783623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.783582    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.785096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.785073    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-envoy" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:47.785298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.785273    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-podinfo" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:47.785373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.785306    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-token" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:47.785373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.785332    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-kube-api-access-wkx54" pod "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" (UID: "ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5"). InnerVolumeSpecName "kube-api-access-wkx54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:47.884846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.884808    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.884846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.884840    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkx54\" (UniqueName: \"kubernetes.io/projected/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-kube-api-access-wkx54\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.885033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.884856    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.885033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.884870    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:47.885033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:47.884882    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:48.451925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.451890    5503 generic.go:356] "Generic (PLEG): container finished" podID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerID="97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e" exitCode=0
Oct 30 22:18:48.452131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.451950    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"
Oct 30 22:18:48.452131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.451973    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" event={"ID":"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5","Type":"ContainerDied","Data":"97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e"}
Oct 30 22:18:48.452131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.452026    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5" event={"ID":"ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5","Type":"ContainerDied","Data":"da30d9254eea35723a2c89bc622eac7a71420802cc716861ea2fa95b11b5fb89"}
Oct 30 22:18:48.452131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.452047    5503 scope.go:122] "RemoveContainer" containerID="97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e"
Oct 30 22:18:48.461599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.461579    5503 scope.go:122] "RemoveContainer" containerID="97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e"
Oct 30 22:18:48.461911 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:18:48.461885    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e\": container with ID starting with 97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e not found: ID does not exist" containerID="97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e"
Oct 30 22:18:48.461982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.461921    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e"} err="failed to get container status \"97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e\": rpc error: code = NotFound desc = could not find container \"97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e\": container with ID starting with 97c35624e8b7c6626d0f36240f9659bb0bbe8c3db95cb76774d0e1f6d25a473e not found: ID does not exist"
Oct 30 22:18:48.679992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.679960    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"]
Oct 30 22:18:48.721782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:48.721693    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-l4cr5"]
Oct 30 22:18:49.634517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:49.634479    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:49.788899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:49.788866    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" path="/var/lib/kubelet/pods/ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5/volumes"
Oct 30 22:18:50.098379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:50.098345    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:52.601121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:52.601090    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:57.120367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:18:57.120330    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:18:58.098965 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:18:58.098924    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:19:01.603386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:01.603354    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:04.565112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:04.565079    5503 scope.go:122] "RemoveContainer" containerID="dac4a2349e0abdced608e59259b4c9208257807d780fdb40e57fc7fec8342427"
Oct 30 22:19:04.575362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:04.575341    5503 scope.go:122] "RemoveContainer" containerID="4232531bd6fb498814fc067387033d00257fc2b81d94cf0e49d474ff1a2d158b"
Oct 30 22:19:04.583534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:04.583516    5503 scope.go:122] "RemoveContainer" containerID="1f6c8b99ee0d7737587b56535b7847e7590d059830e27de99610e2e73d50b338"
Oct 30 22:19:04.591771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:04.591750    5503 scope.go:122] "RemoveContainer" containerID="c9f635e299f9824c103a2d50f537f1b9259b7fdb9ef48a35faefcec9e2eafa7b"
Oct 30 22:19:06.119799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:06.119682    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:06.277590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:06.277547    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:06.429456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:06.429428    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:09.599700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:09.599671    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:19:10.205282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.205250    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"]
Oct 30 22:19:10.205726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.205712    5503 cpu_manager.go:414] "containerMap: removing container" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy"
Oct 30 22:19:10.205777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.205727    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy"
Oct 30 22:19:10.205777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.205745    5503 cpu_manager.go:414] "containerMap: removing container" podUID="ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" containerName="revoke-keys"
Oct 30 22:19:10.205777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.205751    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" containerName="revoke-keys"
Oct 30 22:19:10.205874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.205834    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9d3ec8-1b6c-4181-9c3a-4fbc27b502f5" containerName="istio-proxy"
Oct 30 22:19:10.205874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.205846    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" containerName="revoke-keys"
Oct 30 22:19:10.209153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.209138    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.263203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.263170    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"]
Oct 30 22:19:10.292805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.292771    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc88\" (UniqueName: \"kubernetes.io/projected/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-kube-api-access-8cc88\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.292956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.292898    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.292956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.292938    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.311817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.311788    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:10.311952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.311826    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:10.312078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.312066    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:19:10.313985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.313970    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:10.394072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.394042    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc88\" (UniqueName: \"kubernetes.io/projected/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-kube-api-access-8cc88\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.394231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.394099    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.394231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.394129    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.394838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.394812    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.396477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.396454    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.456776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.456702    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc88\" (UniqueName: \"kubernetes.io/projected/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-kube-api-access-8cc88\") pod \"maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.521310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.521272    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:10.860507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.860429    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:10.889190 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:19:10.889160    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827b19b5_56ad_44d3_b2cf_9c3b53f60e23.slice/crio-db8c25c9cccccdf695826f977e1cf1a6c31ee58de152a40c6abd4fa56ec24c95 WatchSource:0}: Error finding container db8c25c9cccccdf695826f977e1cf1a6c31ee58de152a40c6abd4fa56ec24c95: Status 404 returned error can't find the container with id db8c25c9cccccdf695826f977e1cf1a6c31ee58de152a40c6abd4fa56ec24c95
Oct 30 22:19:10.892139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:10.892121    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"]
Oct 30 22:19:11.612961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:11.612919    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq" event={"ID":"827b19b5-56ad-44d3-b2cf-9c3b53f60e23","Type":"ContainerStarted","Data":"0d86cdf5a9d68ef4f6b2edaf2e6786e675d3df483a6882b1ac9842ddac3f0290"}
Oct 30 22:19:11.613172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:11.612984    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq" event={"ID":"827b19b5-56ad-44d3-b2cf-9c3b53f60e23","Type":"ContainerStarted","Data":"db8c25c9cccccdf695826f977e1cf1a6c31ee58de152a40c6abd4fa56ec24c95"}
Oct 30 22:19:11.763568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:11.763516    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq" podStartSLOduration=1.763501569 podStartE2EDuration="1.763501569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:11.763454287 +0000 UTC m=+1928.098031055" watchObservedRunningTime="2026-10-30 22:19:11.763501569 +0000 UTC m=+1928.098078337"
Oct 30 22:19:12.620175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:12.620141    5503 generic.go:356] "Generic (PLEG): container finished" podID="827b19b5-56ad-44d3-b2cf-9c3b53f60e23" containerID="0d86cdf5a9d68ef4f6b2edaf2e6786e675d3df483a6882b1ac9842ddac3f0290" exitCode=0
Oct 30 22:19:12.620618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:12.620197    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq" event={"ID":"827b19b5-56ad-44d3-b2cf-9c3b53f60e23","Type":"ContainerDied","Data":"0d86cdf5a9d68ef4f6b2edaf2e6786e675d3df483a6882b1ac9842ddac3f0290"}
Oct 30 22:19:13.768090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.768062    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:13.837436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.837406    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-xks-ca\") pod \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") "
Oct 30 22:19:13.837436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.837438    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-openshift-ca\") pod \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") "
Oct 30 22:19:13.837702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.837492    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-kube-api-access-8cc88\" (UniqueName: \"kubernetes.io/projected/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-kube-api-access-8cc88\") pod \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\" (UID: \"827b19b5-56ad-44d3-b2cf-9c3b53f60e23\") "
Oct 30 22:19:13.837930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.837904    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-openshift-ca" pod "827b19b5-56ad-44d3-b2cf-9c3b53f60e23" (UID: "827b19b5-56ad-44d3-b2cf-9c3b53f60e23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:13.839619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.839596    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-xks-ca" pod "827b19b5-56ad-44d3-b2cf-9c3b53f60e23" (UID: "827b19b5-56ad-44d3-b2cf-9c3b53f60e23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:13.839792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.839766    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-kube-api-access-8cc88" pod "827b19b5-56ad-44d3-b2cf-9c3b53f60e23" (UID: "827b19b5-56ad-44d3-b2cf-9c3b53f60e23"). InnerVolumeSpecName "kube-api-access-8cc88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:13.939416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.939385    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:13.939416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.939412    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:13.939416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:13.939423    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cc88\" (UniqueName: \"kubernetes.io/projected/827b19b5-56ad-44d3-b2cf-9c3b53f60e23-kube-api-access-8cc88\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:14.632504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:14.632476    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"
Oct 30 22:19:14.632735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:14.632504    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq" event={"ID":"827b19b5-56ad-44d3-b2cf-9c3b53f60e23","Type":"ContainerDied","Data":"db8c25c9cccccdf695826f977e1cf1a6c31ee58de152a40c6abd4fa56ec24c95"}
Oct 30 22:19:14.632735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:14.632574    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db8c25c9cccccdf695826f977e1cf1a6c31ee58de152a40c6abd4fa56ec24c95"
Oct 30 22:19:15.635266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:15.635233    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:19.567131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:19.567076    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.064754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:20.064723    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.219892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:20.219862    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.649136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:20.649106    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:20.829147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:20.829112    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.156790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:21.156759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.336419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:21.336381    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.502508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:21.502425    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:21.701986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:21.701948    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"]
Oct 30 22:19:21.754003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:21.753913    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad7da0-xzthg"]
Oct 30 22:19:21.797401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:21.797365    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad43b70d-bcb0-4e9e-8e33-248d53f4ee94" path="/var/lib/kubelet/pods/ad43b70d-bcb0-4e9e-8e33-248d53f4ee94/volumes"
Oct 30 22:19:21.927874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:21.927841    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:22.224537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:22.224508    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:22.478862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:22.478736    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:22.703347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:22.703317    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:22.975708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:22.975605    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:23.343342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:23.343259    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:23.520374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:23.520339    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:23.686869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:23.686713    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"]
Oct 30 22:19:23.727658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:23.727320    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-04d404-mmgbq"]
Oct 30 22:19:23.789372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:23.789337    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827b19b5-56ad-44d3-b2cf-9c3b53f60e23" path="/var/lib/kubelet/pods/827b19b5-56ad-44d3-b2cf-9c3b53f60e23/volumes"
Oct 30 22:19:24.042229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:24.042142    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:26.630990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:26.630952    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:27.155900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:27.155866    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:27.666247 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:27.666213    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:27.846550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:27.846518    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:28.099545 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:19:28.099457    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:19:28.710890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:28.710855    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:28.864318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:28.864285    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:29.032276 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:29.032163    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:29.309914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:29.309811    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:29.928419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:29.928378    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:30.693856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.693820    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"]
Oct 30 22:19:30.694333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.694317    5503 cpu_manager.go:414] "containerMap: removing container" podUID="827b19b5-56ad-44d3-b2cf-9c3b53f60e23" containerName="revoke-keys"
Oct 30 22:19:30.694382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.694336    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827b19b5-56ad-44d3-b2cf-9c3b53f60e23" containerName="revoke-keys"
Oct 30 22:19:30.694431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.694420    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="827b19b5-56ad-44d3-b2cf-9c3b53f60e23" containerName="revoke-keys"
Oct 30 22:19:30.697810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.697795    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.708767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.708746    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.708856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.708793    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.708975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.708962    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9d2\" (UniqueName: \"kubernetes.io/projected/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-kube-api-access-sr9d2\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.753719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.753694    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"]
Oct 30 22:19:30.801280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.801256    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:19:30.801431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.801288    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:19:30.801431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.801302    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:30.802001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.801987    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:19:30.810106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.810083    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9d2\" (UniqueName: \"kubernetes.io/projected/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-kube-api-access-sr9d2\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.810222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.810190    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.810284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.810238    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.810858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.810831    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.812574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.812551    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.876189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.876154    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9d2\" (UniqueName: \"kubernetes.io/projected/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-kube-api-access-sr9d2\") pod \"maas-api-revoke-sub-686cb19f-km9m4\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:30.908894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:30.908857    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:31.023185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.023097    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:31.104359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.104330    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:31.275745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.274779    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:31.446356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.446324    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"]
Oct 30 22:19:31.478529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.478504    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"]
Oct 30 22:19:31.484182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.484165    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:31.516721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.516693    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:31.537939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.537890    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7350bc\""
Oct 30 22:19:31.538232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.538217    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7350bc-dockercfg-mxsmz\""
Oct 30 22:19:31.551386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.551362    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"]
Oct 30 22:19:31.616658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.616349    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898932a1-57bb-4391-8c52-a6e5cde990d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:31.616658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.616414    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpcpw\" (UniqueName: \"kubernetes.io/projected/898932a1-57bb-4391-8c52-a6e5cde990d7-kube-api-access-fpcpw\") pod \"payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:31.718277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.718243    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898932a1-57bb-4391-8c52-a6e5cde990d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:31.718462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.718291    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpcpw\" (UniqueName: \"kubernetes.io/projected/898932a1-57bb-4391-8c52-a6e5cde990d7-kube-api-access-fpcpw\") pod \"payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:31.718938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.718916    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898932a1-57bb-4391-8c52-a6e5cde990d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:31.788318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.788245    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4" event={"ID":"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2","Type":"ContainerStarted","Data":"3f5ff32756ee79f7ce4893564fa4f353d2d6376d4652354c1e1778c91760d92a"}
Oct 30 22:19:31.790193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.790173    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpcpw\" (UniqueName: \"kubernetes.io/projected/898932a1-57bb-4391-8c52-a6e5cde990d7-kube-api-access-fpcpw\") pod \"payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:31.816044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:31.816016    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:32.193354 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:19:32.193313    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898932a1_57bb_4391_8c52_a6e5cde990d7.slice/crio-75e12408c3aa803d6b63e2da2fe92fc664a69deeb5870aca8a511212794dc4a2 WatchSource:0}: Error finding container 75e12408c3aa803d6b63e2da2fe92fc664a69deeb5870aca8a511212794dc4a2: Status 404 returned error can't find the container with id 75e12408c3aa803d6b63e2da2fe92fc664a69deeb5870aca8a511212794dc4a2
Oct 30 22:19:32.195902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.195874    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"]
Oct 30 22:19:32.377475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.377443    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"]
Oct 30 22:19:32.381363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.381345    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.437864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.437832    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"]
Oct 30 22:19:32.526500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.526405    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.526500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.526453    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.526817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.526595    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbbl\" (UniqueName: \"kubernetes.io/projected/aacda461-b45d-411c-8f92-e705cf6be935-kube-api-access-gsbbl\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.627680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.627610    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.627865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.627694    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.627865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.627754    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbbl\" (UniqueName: \"kubernetes.io/projected/aacda461-b45d-411c-8f92-e705cf6be935-kube-api-access-gsbbl\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.628239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.628217    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.630048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.630025    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.690233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.690198    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbbl\" (UniqueName: \"kubernetes.io/projected/aacda461-b45d-411c-8f92-e705cf6be935-kube-api-access-gsbbl\") pod \"maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.703043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.703019    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:32.789989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.789914    5503 generic.go:356] "Generic (PLEG): container finished" podID="3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" containerID="08c8d700e57fd9bafc03d12a07c011b8de075c922c7b0e235dfca76be8501c60" exitCode=0
Oct 30 22:19:32.790158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.789999    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4" event={"ID":"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2","Type":"ContainerDied","Data":"08c8d700e57fd9bafc03d12a07c011b8de075c922c7b0e235dfca76be8501c60"}
Oct 30 22:19:32.791942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.791910    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv" event={"ID":"898932a1-57bb-4391-8c52-a6e5cde990d7","Type":"ContainerStarted","Data":"462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045"}
Oct 30 22:19:32.792151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.791964    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv" event={"ID":"898932a1-57bb-4391-8c52-a6e5cde990d7","Type":"ContainerStarted","Data":"75e12408c3aa803d6b63e2da2fe92fc664a69deeb5870aca8a511212794dc4a2"}
Oct 30 22:19:32.792546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:32.792523    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:33.044858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.044749    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv" podStartSLOduration=2.044735718 podStartE2EDuration="2.044735718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:33.042476794 +0000 UTC m=+1949.377053563" watchObservedRunningTime="2026-10-30 22:19:33.044735718 +0000 UTC m=+1949.379312485"
Oct 30 22:19:33.071417 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:19:33.071379    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacda461_b45d_411c_8f92_e705cf6be935.slice/crio-2eefa39810c62759c2cb9a506393b130fe73f147b38327265bff10f383a96867 WatchSource:0}: Error finding container 2eefa39810c62759c2cb9a506393b130fe73f147b38327265bff10f383a96867: Status 404 returned error can't find the container with id 2eefa39810c62759c2cb9a506393b130fe73f147b38327265bff10f383a96867
Oct 30 22:19:33.075265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.075237    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"]
Oct 30 22:19:33.689231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.689192    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:33.798478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.798448    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p" event={"ID":"aacda461-b45d-411c-8f92-e705cf6be935","Type":"ContainerStarted","Data":"8dac9e8169e8fc1b7bdb88988393dc58a4192a85c491110d1ebe710be22fbc5a"}
Oct 30 22:19:33.798660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.798490    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p" event={"ID":"aacda461-b45d-411c-8f92-e705cf6be935","Type":"ContainerStarted","Data":"2eefa39810c62759c2cb9a506393b130fe73f147b38327265bff10f383a96867"}
Oct 30 22:19:33.840623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.840589    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:33.956875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.956846    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:33.958928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:33.958823    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p" podStartSLOduration=1.958807666 podStartE2EDuration="1.958807666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:33.956773775 +0000 UTC m=+1950.291350556" watchObservedRunningTime="2026-10-30 22:19:33.958807666 +0000 UTC m=+1950.293384439"
Oct 30 22:19:34.144723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.144690    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-openshift-ca\") pod \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") "
Oct 30 22:19:34.144905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.144741    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-kube-api-access-sr9d2\" (UniqueName: \"kubernetes.io/projected/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-kube-api-access-sr9d2\") pod \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") "
Oct 30 22:19:34.144905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.144785    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-xks-ca\") pod \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\" (UID: \"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2\") "
Oct 30 22:19:34.145065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.145042    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-openshift-ca" pod "3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" (UID: "3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:34.145268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.145250    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.146873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.146848    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-kube-api-access-sr9d2" pod "3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" (UID: "3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2"). InnerVolumeSpecName "kube-api-access-sr9d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:34.146980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.146885    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-xks-ca" pod "3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" (UID: "3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:34.246968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.246881    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9d2\" (UniqueName: \"kubernetes.io/projected/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-kube-api-access-sr9d2\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.246968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.246907    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:34.805537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.805505    5503 generic.go:356] "Generic (PLEG): container finished" podID="aacda461-b45d-411c-8f92-e705cf6be935" containerID="8dac9e8169e8fc1b7bdb88988393dc58a4192a85c491110d1ebe710be22fbc5a" exitCode=0
Oct 30 22:19:34.806025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.805593    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p" event={"ID":"aacda461-b45d-411c-8f92-e705cf6be935","Type":"ContainerDied","Data":"8dac9e8169e8fc1b7bdb88988393dc58a4192a85c491110d1ebe710be22fbc5a"}
Oct 30 22:19:34.807005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.806991    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"
Oct 30 22:19:34.807005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.806994    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4" event={"ID":"3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2","Type":"ContainerDied","Data":"3f5ff32756ee79f7ce4893564fa4f353d2d6376d4652354c1e1778c91760d92a"}
Oct 30 22:19:34.807142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.807035    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f5ff32756ee79f7ce4893564fa4f353d2d6376d4652354c1e1778c91760d92a"
Oct 30 22:19:34.850198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:34.850169    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:35.356987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.356949    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"]
Oct 30 22:19:35.357507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.357491    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" containerName="revoke-keys"
Oct 30 22:19:35.357554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.357509    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" containerName="revoke-keys"
Oct 30 22:19:35.357613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.357603    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" containerName="revoke-keys"
Oct 30 22:19:35.361042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.361025    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.416023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.415992    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"]
Oct 30 22:19:35.461891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.461860    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnztw\" (UniqueName: \"kubernetes.io/projected/90d4e1bb-aedd-4215-9a35-b0913584b75a-kube-api-access-nnztw\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.462027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.461901    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.462080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.462019    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.563693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.563661    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.563858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.563839    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnztw\" (UniqueName: \"kubernetes.io/projected/90d4e1bb-aedd-4215-9a35-b0913584b75a-kube-api-access-nnztw\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.563908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.563887    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.564351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.564327    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.566468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.566441    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.626113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.626050    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnztw\" (UniqueName: \"kubernetes.io/projected/90d4e1bb-aedd-4215-9a35-b0913584b75a-kube-api-access-nnztw\") pod \"maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.674008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.673983    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:35.951226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.951200    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:35.968472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.968444    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-openshift-ca\") pod \"aacda461-b45d-411c-8f92-e705cf6be935\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") "
Oct 30 22:19:35.968589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.968494    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-xks-ca\") pod \"aacda461-b45d-411c-8f92-e705cf6be935\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") "
Oct 30 22:19:35.968589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.968577    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacda461-b45d-411c-8f92-e705cf6be935-kube-api-access-gsbbl\" (UniqueName: \"kubernetes.io/projected/aacda461-b45d-411c-8f92-e705cf6be935-kube-api-access-gsbbl\") pod \"aacda461-b45d-411c-8f92-e705cf6be935\" (UID: \"aacda461-b45d-411c-8f92-e705cf6be935\") "
Oct 30 22:19:35.968792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.968770    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-openshift-ca" pod "aacda461-b45d-411c-8f92-e705cf6be935" (UID: "aacda461-b45d-411c-8f92-e705cf6be935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:35.968960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.968942    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:35.970475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.970457    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-xks-ca" pod "aacda461-b45d-411c-8f92-e705cf6be935" (UID: "aacda461-b45d-411c-8f92-e705cf6be935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:35.970542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:35.970522    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacda461-b45d-411c-8f92-e705cf6be935-kube-api-access-gsbbl" pod "aacda461-b45d-411c-8f92-e705cf6be935" (UID: "aacda461-b45d-411c-8f92-e705cf6be935"). InnerVolumeSpecName "kube-api-access-gsbbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:36.044000 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:19:36.043975    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d4e1bb_aedd_4215_9a35_b0913584b75a.slice/crio-d4850c312ea739744e16551a5c5459bdc6bd62fbbe7a8f3bcc338710e4999be9 WatchSource:0}: Error finding container d4850c312ea739744e16551a5c5459bdc6bd62fbbe7a8f3bcc338710e4999be9: Status 404 returned error can't find the container with id d4850c312ea739744e16551a5c5459bdc6bd62fbbe7a8f3bcc338710e4999be9
Oct 30 22:19:36.046572 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.046544    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"]
Oct 30 22:19:36.070268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.069992    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacda461-b45d-411c-8f92-e705cf6be935-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:36.070268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.070020    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsbbl\" (UniqueName: \"kubernetes.io/projected/aacda461-b45d-411c-8f92-e705cf6be935-kube-api-access-gsbbl\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:36.819256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.819229    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"
Oct 30 22:19:36.819430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.819260    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p" event={"ID":"aacda461-b45d-411c-8f92-e705cf6be935","Type":"ContainerDied","Data":"2eefa39810c62759c2cb9a506393b130fe73f147b38327265bff10f383a96867"}
Oct 30 22:19:36.819430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.819311    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eefa39810c62759c2cb9a506393b130fe73f147b38327265bff10f383a96867"
Oct 30 22:19:36.821085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.821057    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5" event={"ID":"90d4e1bb-aedd-4215-9a35-b0913584b75a","Type":"ContainerStarted","Data":"2166d98296981decdfac4666d034753d960ad6981170eb8f30e67b41fdfa0be4"}
Oct 30 22:19:36.821228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.821105    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5" event={"ID":"90d4e1bb-aedd-4215-9a35-b0913584b75a","Type":"ContainerStarted","Data":"d4850c312ea739744e16551a5c5459bdc6bd62fbbe7a8f3bcc338710e4999be9"}
Oct 30 22:19:36.940745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:36.940699    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5" podStartSLOduration=1.9406845499999998 podStartE2EDuration="1.94068455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:36.938542033 +0000 UTC m=+1953.273118802" watchObservedRunningTime="2026-10-30 22:19:36.94068455 +0000 UTC m=+1953.275261319"
Oct 30 22:19:37.828818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:37.828785    5503 generic.go:356] "Generic (PLEG): container finished" podID="90d4e1bb-aedd-4215-9a35-b0913584b75a" containerID="2166d98296981decdfac4666d034753d960ad6981170eb8f30e67b41fdfa0be4" exitCode=0
Oct 30 22:19:37.829227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:37.828864    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5" event={"ID":"90d4e1bb-aedd-4215-9a35-b0913584b75a","Type":"ContainerDied","Data":"2166d98296981decdfac4666d034753d960ad6981170eb8f30e67b41fdfa0be4"}
Oct 30 22:19:38.981063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:38.981034    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:39.004483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.004457    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-openshift-ca\") pod \"90d4e1bb-aedd-4215-9a35-b0913584b75a\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") "
Oct 30 22:19:39.004658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.004629    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-xks-ca\") pod \"90d4e1bb-aedd-4215-9a35-b0913584b75a\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") "
Oct 30 22:19:39.004740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.004681    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d4e1bb-aedd-4215-9a35-b0913584b75a-kube-api-access-nnztw\" (UniqueName: \"kubernetes.io/projected/90d4e1bb-aedd-4215-9a35-b0913584b75a-kube-api-access-nnztw\") pod \"90d4e1bb-aedd-4215-9a35-b0913584b75a\" (UID: \"90d4e1bb-aedd-4215-9a35-b0913584b75a\") "
Oct 30 22:19:39.004880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.004853    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-openshift-ca" pod "90d4e1bb-aedd-4215-9a35-b0913584b75a" (UID: "90d4e1bb-aedd-4215-9a35-b0913584b75a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:39.005065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.005048    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:39.006580 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.006557    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-xks-ca" pod "90d4e1bb-aedd-4215-9a35-b0913584b75a" (UID: "90d4e1bb-aedd-4215-9a35-b0913584b75a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:39.006700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.006683    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d4e1bb-aedd-4215-9a35-b0913584b75a-kube-api-access-nnztw" pod "90d4e1bb-aedd-4215-9a35-b0913584b75a" (UID: "90d4e1bb-aedd-4215-9a35-b0913584b75a"). InnerVolumeSpecName "kube-api-access-nnztw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:39.113571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.113494    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90d4e1bb-aedd-4215-9a35-b0913584b75a-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:39.113571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.113525    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnztw\" (UniqueName: \"kubernetes.io/projected/90d4e1bb-aedd-4215-9a35-b0913584b75a-kube-api-access-nnztw\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:39.842317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.842289    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"
Oct 30 22:19:39.842317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.842308    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5" event={"ID":"90d4e1bb-aedd-4215-9a35-b0913584b75a","Type":"ContainerDied","Data":"d4850c312ea739744e16551a5c5459bdc6bd62fbbe7a8f3bcc338710e4999be9"}
Oct 30 22:19:39.842490 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.842346    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4850c312ea739744e16551a5c5459bdc6bd62fbbe7a8f3bcc338710e4999be9"
Oct 30 22:19:39.909766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:39.909726    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:41.478820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:41.478781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:42.197539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:42.197499    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:19:44.810306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:44.810276    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:19:46.023482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:46.023449    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:47.676325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:47.676292    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:50.571388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:50.571323    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:55.135408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:55.135366    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:19:58.097994 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:19:58.097955    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:19:59.626421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:19:59.626382    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:04.137580 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:04.137550    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:04.637293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:04.637262    5503 scope.go:122] "RemoveContainer" containerID="8a5dd2bda89dcb6058f6913eaf9fa003f2ab11a43707ccd5122e99a3164e6dc8"
Oct 30 22:20:04.649134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:04.649116    5503 scope.go:122] "RemoveContainer" containerID="58a281415e2ab71cd87d9f9dabe4d6890aebc1caaa5fd155d774bf2cec77eb9d"
Oct 30 22:20:05.685079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:05.684660    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:06.052237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:06.052157    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"]
Oct 30 22:20:06.182715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:06.182679    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"]
Oct 30 22:20:06.524286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:06.524248    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:06.695600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:06.695511    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:07.482669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:07.465516    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:08.027919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:08.025211    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:09.872919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:09.872888    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:11.053164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.053128    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx" podUID="ed201742-12c7-4fde-8c56-705a8de721ca" containerName="payload-processing" containerID="cri-o://023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847" gracePeriod=25
Oct 30 22:20:11.183480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.183442    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t" podUID="970a9597-129f-4a5e-a976-7092f7e07590" containerName="payload-pre-processing" containerID="cri-o://edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474" gracePeriod=25
Oct 30 22:20:11.326948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.326916    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:20:11.419812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.419779    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed201742-12c7-4fde-8c56-705a8de721ca-kube-api-access-ls47k\" (UniqueName: \"kubernetes.io/projected/ed201742-12c7-4fde-8c56-705a8de721ca-kube-api-access-ls47k\") pod \"ed201742-12c7-4fde-8c56-705a8de721ca\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") "
Oct 30 22:20:11.419974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.419858    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed201742-12c7-4fde-8c56-705a8de721ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed201742-12c7-4fde-8c56-705a8de721ca-plugins-config-volume\") pod \"ed201742-12c7-4fde-8c56-705a8de721ca\" (UID: \"ed201742-12c7-4fde-8c56-705a8de721ca\") "
Oct 30 22:20:11.420174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.420150    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed201742-12c7-4fde-8c56-705a8de721ca-plugins-config-volume" pod "ed201742-12c7-4fde-8c56-705a8de721ca" (UID: "ed201742-12c7-4fde-8c56-705a8de721ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:11.420321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.420308    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed201742-12c7-4fde-8c56-705a8de721ca-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:11.421973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.421947    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed201742-12c7-4fde-8c56-705a8de721ca-kube-api-access-ls47k" pod "ed201742-12c7-4fde-8c56-705a8de721ca" (UID: "ed201742-12c7-4fde-8c56-705a8de721ca"). InnerVolumeSpecName "kube-api-access-ls47k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:11.440952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.440927    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:20:11.521202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.521174    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970a9597-129f-4a5e-a976-7092f7e07590-kube-api-access-9vdzd\" (UniqueName: \"kubernetes.io/projected/970a9597-129f-4a5e-a976-7092f7e07590-kube-api-access-9vdzd\") pod \"970a9597-129f-4a5e-a976-7092f7e07590\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") "
Oct 30 22:20:11.521407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.521348    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970a9597-129f-4a5e-a976-7092f7e07590-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970a9597-129f-4a5e-a976-7092f7e07590-plugins-config-volume\") pod \"970a9597-129f-4a5e-a976-7092f7e07590\" (UID: \"970a9597-129f-4a5e-a976-7092f7e07590\") "
Oct 30 22:20:11.521659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.521626    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls47k\" (UniqueName: \"kubernetes.io/projected/ed201742-12c7-4fde-8c56-705a8de721ca-kube-api-access-ls47k\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:11.521797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.521671    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970a9597-129f-4a5e-a976-7092f7e07590-plugins-config-volume" pod "970a9597-129f-4a5e-a976-7092f7e07590" (UID: "970a9597-129f-4a5e-a976-7092f7e07590"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:11.523557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.523529    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970a9597-129f-4a5e-a976-7092f7e07590-kube-api-access-9vdzd" pod "970a9597-129f-4a5e-a976-7092f7e07590" (UID: "970a9597-129f-4a5e-a976-7092f7e07590"). InnerVolumeSpecName "kube-api-access-9vdzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:11.623325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.623242    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/970a9597-129f-4a5e-a976-7092f7e07590-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:11.623325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:11.623271    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vdzd\" (UniqueName: \"kubernetes.io/projected/970a9597-129f-4a5e-a976-7092f7e07590-kube-api-access-9vdzd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:20:12.051699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.051656    5503 generic.go:356] "Generic (PLEG): container finished" podID="970a9597-129f-4a5e-a976-7092f7e07590" containerID="edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474" exitCode=0
Oct 30 22:20:12.051873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.051700    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"
Oct 30 22:20:12.051873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.051733    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t" event={"ID":"970a9597-129f-4a5e-a976-7092f7e07590","Type":"ContainerDied","Data":"edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474"}
Oct 30 22:20:12.051873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.051782    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t" event={"ID":"970a9597-129f-4a5e-a976-7092f7e07590","Type":"ContainerDied","Data":"da63d2c1151ade42910b3d6f86c1cf4d0be6fe9ddd41410c94f64600d28c145e"}
Oct 30 22:20:12.051873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.051813    5503 scope.go:122] "RemoveContainer" containerID="edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474"
Oct 30 22:20:12.053170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.053145    5503 generic.go:356] "Generic (PLEG): container finished" podID="ed201742-12c7-4fde-8c56-705a8de721ca" containerID="023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847" exitCode=0
Oct 30 22:20:12.053293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.053203    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"
Oct 30 22:20:12.053293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.053229    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx" event={"ID":"ed201742-12c7-4fde-8c56-705a8de721ca","Type":"ContainerDied","Data":"023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847"}
Oct 30 22:20:12.053293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.053288    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx" event={"ID":"ed201742-12c7-4fde-8c56-705a8de721ca","Type":"ContainerDied","Data":"d191d8deae94d8a8f0626aac08f67a2ea4604e19b7df82df051433fd9c81ca2e"}
Oct 30 22:20:12.061883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.061713    5503 scope.go:122] "RemoveContainer" containerID="edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474"
Oct 30 22:20:12.062009 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:20:12.061991    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474\": container with ID starting with edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474 not found: ID does not exist" containerID="edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474"
Oct 30 22:20:12.062050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.062018    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474"} err="failed to get container status \"edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474\": rpc error: code = NotFound desc = could not find container \"edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474\": container with ID starting with edfe2f4b08a6c787f963d1e2351c07a9c0cf5da2197d27c8c8235e6678883474 not found: ID does not exist"
Oct 30 22:20:12.062050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.062035    5503 scope.go:122] "RemoveContainer" containerID="023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847"
Oct 30 22:20:12.074174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.074146    5503 scope.go:122] "RemoveContainer" containerID="023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847"
Oct 30 22:20:12.074421 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:20:12.074404    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847\": container with ID starting with 023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847 not found: ID does not exist" containerID="023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847"
Oct 30 22:20:12.074471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.074427    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847"} err="failed to get container status \"023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847\": rpc error: code = NotFound desc = could not find container \"023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847\": container with ID starting with 023adfec8e586ccbfb27752ad0f4d615afb8713f1a37b249d13cd84239e67847 not found: ID does not exist"
Oct 30 22:20:12.220227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.220190    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"]
Oct 30 22:20:12.270854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.270819    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0e90fe-5fbb94d8bc-5tmbx"]
Oct 30 22:20:12.338519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.338437    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"]
Oct 30 22:20:12.342842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.342822    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0e90fe-5c6c4d9c5-q979t"]
Oct 30 22:20:12.907246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:12.907168    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.081870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:13.081826    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.249821 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:13.249746    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.422675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:13.420579    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:13.789128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:13.789098    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970a9597-129f-4a5e-a976-7092f7e07590" path="/var/lib/kubelet/pods/970a9597-129f-4a5e-a976-7092f7e07590/volumes"
Oct 30 22:20:13.789475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:13.789462    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed201742-12c7-4fde-8c56-705a8de721ca" path="/var/lib/kubelet/pods/ed201742-12c7-4fde-8c56-705a8de721ca/volumes"
Oct 30 22:20:13.885685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:13.885656    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:15.960877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:15.960843    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:16.529334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:16.529294    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:16.683330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:16.683295    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:17.684540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:17.684505    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:17.882184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:17.882153    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:18.085906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.085822    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:18.237895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.237864    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:18.517139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.517103    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:18.707684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.707621    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:18.759420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.759385    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"]
Oct 30 22:20:18.760922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.760895    5503 cpu_manager.go:414] "containerMap: removing container" podUID="ed201742-12c7-4fde-8c56-705a8de721ca" containerName="payload-processing"
Oct 30 22:20:18.761099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761068    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed201742-12c7-4fde-8c56-705a8de721ca" containerName="payload-processing"
Oct 30 22:20:18.761252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761118    5503 cpu_manager.go:414] "containerMap: removing container" podUID="aacda461-b45d-411c-8f92-e705cf6be935" containerName="revoke-keys"
Oct 30 22:20:18.761252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761127    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacda461-b45d-411c-8f92-e705cf6be935" containerName="revoke-keys"
Oct 30 22:20:18.761252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761221    5503 cpu_manager.go:414] "containerMap: removing container" podUID="970a9597-129f-4a5e-a976-7092f7e07590" containerName="payload-pre-processing"
Oct 30 22:20:18.761252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761230    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970a9597-129f-4a5e-a976-7092f7e07590" containerName="payload-pre-processing"
Oct 30 22:20:18.761488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761282    5503 cpu_manager.go:414] "containerMap: removing container" podUID="90d4e1bb-aedd-4215-9a35-b0913584b75a" containerName="revoke-keys"
Oct 30 22:20:18.761488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761290    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d4e1bb-aedd-4215-9a35-b0913584b75a" containerName="revoke-keys"
Oct 30 22:20:18.761590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761532    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="970a9597-129f-4a5e-a976-7092f7e07590" containerName="payload-pre-processing"
Oct 30 22:20:18.761590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761555    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d4e1bb-aedd-4215-9a35-b0913584b75a" containerName="revoke-keys"
Oct 30 22:20:18.761590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761567    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed201742-12c7-4fde-8c56-705a8de721ca" containerName="payload-processing"
Oct 30 22:20:18.761590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.761584    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacda461-b45d-411c-8f92-e705cf6be935" containerName="revoke-keys"
Oct 30 22:20:18.786763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.775463    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:18.824790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.824757    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"]
Oct 30 22:20:18.856627 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.856582    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:18.856815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.856776    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlgkg\" (UniqueName: \"kubernetes.io/projected/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-kube-api-access-tlgkg\") pod \"payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:18.873168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.873148    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a3357\""
Oct 30 22:20:18.873168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.873155    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a3357-dockercfg-l9cnh\""
Oct 30 22:20:18.957600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.957568    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:18.957766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.957631    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlgkg\" (UniqueName: \"kubernetes.io/projected/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-kube-api-access-tlgkg\") pod \"payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:18.958347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.958325    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:18.966439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:18.966407    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:19.019677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:19.019651    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlgkg\" (UniqueName: \"kubernetes.io/projected/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-kube-api-access-tlgkg\") pod \"payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:19.077955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:19.077880    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"]
Oct 30 22:20:19.081378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:19.081355    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4576d9-8sbb5"]
Oct 30 22:20:19.104312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:19.104294    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:19.477928 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:20:19.477891    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085a50b4_b09e_4f90_b3c8_3b71217cf4ae.slice/crio-452d48231c2a0fbfd16c2641a1c562c3fa44861255fce6a606e627eb5cab6949 WatchSource:0}: Error finding container 452d48231c2a0fbfd16c2641a1c562c3fa44861255fce6a606e627eb5cab6949: Status 404 returned error can't find the container with id 452d48231c2a0fbfd16c2641a1c562c3fa44861255fce6a606e627eb5cab6949
Oct 30 22:20:19.480264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:19.480238    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"]
Oct 30 22:20:19.520944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:19.520916    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:19.791457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:19.791372    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d4e1bb-aedd-4215-9a35-b0913584b75a" path="/var/lib/kubelet/pods/90d4e1bb-aedd-4215-9a35-b0913584b75a/volumes"
Oct 30 22:20:20.125521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:20.125432    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t" event={"ID":"085a50b4-b09e-4f90-b3c8-3b71217cf4ae","Type":"ContainerStarted","Data":"17e7b889c14977835c53efb06d3e305e2b23e9094ffde04b0dac9bff27415ce2"}
Oct 30 22:20:20.125521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:20.125493    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t" event={"ID":"085a50b4-b09e-4f90-b3c8-3b71217cf4ae","Type":"ContainerStarted","Data":"452d48231c2a0fbfd16c2641a1c562c3fa44861255fce6a606e627eb5cab6949"}
Oct 30 22:20:20.125744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:20.125524    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:20.296238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:20.296184    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t" podStartSLOduration=2.296169631 podStartE2EDuration="2.296169631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:20.243723145 +0000 UTC m=+1996.578299914" watchObservedRunningTime="2026-10-30 22:20:20.296169631 +0000 UTC m=+1996.630746399"
Oct 30 22:20:20.999392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:20.999356    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:22.109841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:22.109805    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:22.862694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:22.862664    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:25.924910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:25.924865    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:26.098747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:26.098718    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:26.301451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:26.301366    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:27.542057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:27.542022    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:27.894986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:27.894907    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:28.097687 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:20:28.097626    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:20:28.580083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:28.580046    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:28.893741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:28.893658    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:31.134815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:31.134778    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:20:31.577736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:31.577703    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.160314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:32.160280    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:32.316769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:32.316733    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:33.173085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:33.173052    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:33.372814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:33.372781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:33.527092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:33.527020    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:33.862333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:33.862252    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:34.119176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:34.119103    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:34.324123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:34.324090    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"]
Oct 30 22:20:34.374851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:34.374784    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8b4b54-clb5p"]
Oct 30 22:20:34.738498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:34.738454    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:35.789147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:35.789115    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacda461-b45d-411c-8f92-e705cf6be935" path="/var/lib/kubelet/pods/aacda461-b45d-411c-8f92-e705cf6be935/volumes"
Oct 30 22:20:36.376282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:36.376251    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:36.558688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:36.558658    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:38.380933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:38.380898    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:41.173218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:41.173183    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:41.553999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:41.553886    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:41.724272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:41.724230    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:43.113569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:43.113532    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:43.282666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:43.282606    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:44.045310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:44.045272    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:44.203387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:44.203306    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:44.926180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:44.926143    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:20:47.844318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:47.844268    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:48.018442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:48.018400    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:48.168559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:48.168524    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:48.329502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:48.329464    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:51.109920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:51.109886    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:51.305681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:51.305650    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:51.861629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:51.861570    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:52.395217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:52.395180    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:54.835757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:54.835722    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:20:58.099086 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:20:58.099046    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:20:59.673659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:20:59.672791    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:04.675438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:04.675409    5503 scope.go:122] "RemoveContainer" containerID="95352813a2922d1a0ac62304ebb285ad8195ecd554892692d26dd8ebf19d572a"
Oct 30 22:21:05.695380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:05.695338    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:05.849335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:05.849294    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:06.008618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:06.008541    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:10.443320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:10.443285    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.169656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.165962    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.698973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.698935    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"]
Oct 30 22:21:12.704559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.704536    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.771748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.771709    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-872f4a-openshift-default-dockercfg-zr6s8\""
Oct 30 22:21:12.822110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.822073    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istiod-ca-cert\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.888614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.888576    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-socket\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.888948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.888913    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-podinfo\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.889191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.889160    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-token\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.889377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.889349    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-data\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.889600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.889582    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-certs\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.889771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.889751    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-kube-api-access-jd4wf\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.889985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.889965    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-credential-socket\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.890151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.890135    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-envoy\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.890817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.824142    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:12.891866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.891847    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"]
Oct 30 22:21:12.991939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.991853    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istiod-ca-cert\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.991939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.991891    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-socket\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.991939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.991913    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-podinfo\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.991939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.991938    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-token\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.991966    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-data\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992020    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-certs\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992047    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-kube-api-access-jd4wf\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992120    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-credential-socket\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992172    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-envoy\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992325    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-socket\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992445    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-data\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992676    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-credential-socket\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992754    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-certs\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.992796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.992769    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istiod-ca-cert\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.994652 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.994619    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-envoy\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:12.994885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:12.994860    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-podinfo\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:13.054437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.054404    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-token\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:13.055014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.054983    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-kube-api-access-jd4wf\") pod \"e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") " pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:13.179902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.179863    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:13.320190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.320076    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:13.747354 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:21:13.747320    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea557cd_e74e_4797_bdbc_bc338f21e2e1.slice/crio-fef32f6a72d53bfadd6b672d96be2b325eb9e63cd6cc1aa46b5bba2b8d26c565 WatchSource:0}: Error finding container fef32f6a72d53bfadd6b672d96be2b325eb9e63cd6cc1aa46b5bba2b8d26c565: Status 404 returned error can't find the container with id fef32f6a72d53bfadd6b672d96be2b325eb9e63cd6cc1aa46b5bba2b8d26c565
Oct 30 22:21:13.749738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.749713    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"]
Oct 30 22:21:13.750114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.750082    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:21:13.750163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.750144    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:21:13.750195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:13.750174    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:21:14.510270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:14.510234    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" event={"ID":"9ea557cd-e74e-4797-bdbc-bc338f21e2e1","Type":"ContainerStarted","Data":"c0d61e2b19da6d43e5555d3678631cbf9e39946d954034a83faca9ac18ab8245"}
Oct 30 22:21:14.510703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:14.510289    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" event={"ID":"9ea557cd-e74e-4797-bdbc-bc338f21e2e1","Type":"ContainerStarted","Data":"fef32f6a72d53bfadd6b672d96be2b325eb9e63cd6cc1aa46b5bba2b8d26c565"}
Oct 30 22:21:14.687034 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:14.686977    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podStartSLOduration=2.686964016 podStartE2EDuration="2.686964016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:14.68431519 +0000 UTC m=+2051.018891959" watchObservedRunningTime="2026-10-30 22:21:14.686964016 +0000 UTC m=+2051.021540785"
Oct 30 22:21:15.040586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:15.040547    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:15.320749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:15.320668    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:15.326257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:15.326229    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:15.516437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:15.516406    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:15.517217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:15.517198    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:21:17.574151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:17.574118    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:17.748497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:17.748465    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:17.901659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:17.901599    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:19.971217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:19.971184    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:20.692356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:20.692322    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:21:21.479408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:21.479379    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:22.248470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.248434    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"]
Oct 30 22:21:22.434331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.434302    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"]
Oct 30 22:21:22.438298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.438279    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.484075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.484008    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:22.492395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.492367    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"]
Oct 30 22:21:22.494070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.494022    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:21:22.494225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.494078    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:21:22.494225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.494180    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:22.494444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.494431    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:21:22.537270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.537181    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.537270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.537249    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tbd\" (UniqueName: \"kubernetes.io/projected/24f6c369-be9d-4cce-83ad-a8fe22e89a37-kube-api-access-d2tbd\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.537487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.537330    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.638394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.638361    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.638591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.638505    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.638591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.638555    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tbd\" (UniqueName: \"kubernetes.io/projected/24f6c369-be9d-4cce-83ad-a8fe22e89a37-kube-api-access-d2tbd\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.639229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.639209    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.640814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.640794    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.643724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.643693    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:22.709819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.705474    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tbd\" (UniqueName: \"kubernetes.io/projected/24f6c369-be9d-4cce-83ad-a8fe22e89a37-kube-api-access-d2tbd\") pod \"maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.752038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.751998    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:22.799745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:22.799647    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:23.124572 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:21:23.124544    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f6c369_be9d_4cce_83ad_a8fe22e89a37.slice/crio-27ed7ad17a7f3ee1ba9fe542b166b5b5adb5d6eafa221aea40c3f75ac15ac1ad WatchSource:0}: Error finding container 27ed7ad17a7f3ee1ba9fe542b166b5b5adb5d6eafa221aea40c3f75ac15ac1ad: Status 404 returned error can't find the container with id 27ed7ad17a7f3ee1ba9fe542b166b5b5adb5d6eafa221aea40c3f75ac15ac1ad
Oct 30 22:21:23.126335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:23.126309    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"]
Oct 30 22:21:23.587138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:23.587100    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw" event={"ID":"24f6c369-be9d-4cce-83ad-a8fe22e89a37","Type":"ContainerStarted","Data":"5ccc6c046af78faeefe07d0f499d25b757dffc30999d2ca7c3fa796b0b00c60e"}
Oct 30 22:21:23.587656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:23.587156    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw" event={"ID":"24f6c369-be9d-4cce-83ad-a8fe22e89a37","Type":"ContainerStarted","Data":"27ed7ad17a7f3ee1ba9fe542b166b5b5adb5d6eafa221aea40c3f75ac15ac1ad"}
Oct 30 22:21:23.706550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:23.706457    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw" podStartSLOduration=1.706442298 podStartE2EDuration="1.706442298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.705774561 +0000 UTC m=+2060.040351331" watchObservedRunningTime="2026-10-30 22:21:23.706442298 +0000 UTC m=+2060.041019116"
Oct 30 22:21:24.594351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:24.594318    5503 generic.go:356] "Generic (PLEG): container finished" podID="24f6c369-be9d-4cce-83ad-a8fe22e89a37" containerID="5ccc6c046af78faeefe07d0f499d25b757dffc30999d2ca7c3fa796b0b00c60e" exitCode=0
Oct 30 22:21:24.594773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:24.594357    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw" event={"ID":"24f6c369-be9d-4cce-83ad-a8fe22e89a37","Type":"ContainerDied","Data":"5ccc6c046af78faeefe07d0f499d25b757dffc30999d2ca7c3fa796b0b00c60e"}
Oct 30 22:21:25.225241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.225203    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:25.386104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.386068    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:25.543375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.543289    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:25.742088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.742063    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:25.919951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.919912    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f6c369-be9d-4cce-83ad-a8fe22e89a37-kube-api-access-d2tbd\" (UniqueName: \"kubernetes.io/projected/24f6c369-be9d-4cce-83ad-a8fe22e89a37-kube-api-access-d2tbd\") pod \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") "
Oct 30 22:21:25.920156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.920058    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-xks-ca\") pod \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") "
Oct 30 22:21:25.920156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.920100    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-openshift-ca\") pod \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\" (UID: \"24f6c369-be9d-4cce-83ad-a8fe22e89a37\") "
Oct 30 22:21:25.920481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.920456    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-openshift-ca" pod "24f6c369-be9d-4cce-83ad-a8fe22e89a37" (UID: "24f6c369-be9d-4cce-83ad-a8fe22e89a37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:25.922071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.922039    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-xks-ca" pod "24f6c369-be9d-4cce-83ad-a8fe22e89a37" (UID: "24f6c369-be9d-4cce-83ad-a8fe22e89a37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:25.922175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:25.922078    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f6c369-be9d-4cce-83ad-a8fe22e89a37-kube-api-access-d2tbd" pod "24f6c369-be9d-4cce-83ad-a8fe22e89a37" (UID: "24f6c369-be9d-4cce-83ad-a8fe22e89a37"). InnerVolumeSpecName "kube-api-access-d2tbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:26.021368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:26.021336    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2tbd\" (UniqueName: \"kubernetes.io/projected/24f6c369-be9d-4cce-83ad-a8fe22e89a37-kube-api-access-d2tbd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:26.021368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:26.021363    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:26.021557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:26.021373    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f6c369-be9d-4cce-83ad-a8fe22e89a37-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:26.607937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:26.607897    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw" event={"ID":"24f6c369-be9d-4cce-83ad-a8fe22e89a37","Type":"ContainerDied","Data":"27ed7ad17a7f3ee1ba9fe542b166b5b5adb5d6eafa221aea40c3f75ac15ac1ad"}
Oct 30 22:21:26.607937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:26.607921    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"
Oct 30 22:21:26.608158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:26.607953    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ed7ad17a7f3ee1ba9fe542b166b5b5adb5d6eafa221aea40c3f75ac15ac1ad"
Oct 30 22:21:27.249273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.249215    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv" podUID="898932a1-57bb-4391-8c52-a6e5cde990d7" containerName="payload-processing" containerID="cri-o://462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045" gracePeriod=25
Oct 30 22:21:27.503594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.503533    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:21:27.535794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.535756    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898932a1-57bb-4391-8c52-a6e5cde990d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898932a1-57bb-4391-8c52-a6e5cde990d7-plugins-config-volume\") pod \"898932a1-57bb-4391-8c52-a6e5cde990d7\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") "
Oct 30 22:21:27.535968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.535834    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898932a1-57bb-4391-8c52-a6e5cde990d7-kube-api-access-fpcpw\" (UniqueName: \"kubernetes.io/projected/898932a1-57bb-4391-8c52-a6e5cde990d7-kube-api-access-fpcpw\") pod \"898932a1-57bb-4391-8c52-a6e5cde990d7\" (UID: \"898932a1-57bb-4391-8c52-a6e5cde990d7\") "
Oct 30 22:21:27.536144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.536121    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898932a1-57bb-4391-8c52-a6e5cde990d7-plugins-config-volume" pod "898932a1-57bb-4391-8c52-a6e5cde990d7" (UID: "898932a1-57bb-4391-8c52-a6e5cde990d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:27.536257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.536241    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/898932a1-57bb-4391-8c52-a6e5cde990d7-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:27.537933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.537911    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898932a1-57bb-4391-8c52-a6e5cde990d7-kube-api-access-fpcpw" pod "898932a1-57bb-4391-8c52-a6e5cde990d7" (UID: "898932a1-57bb-4391-8c52-a6e5cde990d7"). InnerVolumeSpecName "kube-api-access-fpcpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:27.616553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.616520    5503 generic.go:356] "Generic (PLEG): container finished" podID="898932a1-57bb-4391-8c52-a6e5cde990d7" containerID="462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045" exitCode=0
Oct 30 22:21:27.616758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.616576    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"
Oct 30 22:21:27.616758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.616578    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv" event={"ID":"898932a1-57bb-4391-8c52-a6e5cde990d7","Type":"ContainerDied","Data":"462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045"}
Oct 30 22:21:27.616758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.616709    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv" event={"ID":"898932a1-57bb-4391-8c52-a6e5cde990d7","Type":"ContainerDied","Data":"75e12408c3aa803d6b63e2da2fe92fc664a69deeb5870aca8a511212794dc4a2"}
Oct 30 22:21:27.616758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.616733    5503 scope.go:122] "RemoveContainer" containerID="462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045"
Oct 30 22:21:27.628365 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.628344    5503 scope.go:122] "RemoveContainer" containerID="462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045"
Oct 30 22:21:27.628756 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:21:27.628734    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045\": container with ID starting with 462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045 not found: ID does not exist" containerID="462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045"
Oct 30 22:21:27.628865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.628765    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045"} err="failed to get container status \"462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045\": rpc error: code = NotFound desc = could not find container \"462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045\": container with ID starting with 462546292eeae6e55db0a800d22c9e707426b2fa0959719b5ff2690b1a157045 not found: ID does not exist"
Oct 30 22:21:27.637085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.637062    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpcpw\" (UniqueName: \"kubernetes.io/projected/898932a1-57bb-4391-8c52-a6e5cde990d7-kube-api-access-fpcpw\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:27.795455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.795385    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"]
Oct 30 22:21:27.801796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:27.801766    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7350bc-7794bfff7f-zdzqv"]
Oct 30 22:21:28.099294 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:21:28.099204    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:21:29.789181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:29.789147    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898932a1-57bb-4391-8c52-a6e5cde990d7" path="/var/lib/kubelet/pods/898932a1-57bb-4391-8c52-a6e5cde990d7/volumes"
Oct 30 22:21:30.018492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:30.018457    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:34.642829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:34.642793    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:39.193331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:39.193291    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:43.670748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:43.670711    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:43.821903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:43.821871    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:43.977008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:43.976929    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:47.277138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:47.277102    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:21:48.588922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:48.588888    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:54.786220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:54.786184    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:55.405923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:55.405886    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:58.099984 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:21:58.099943    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:21:58.745322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:58.745282    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:59.227017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:59.226971    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:21:59.395195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:21:59.395163    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.232943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:00.232907    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.469336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:00.469301    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.626701 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:00.626600    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:00.706853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:00.706820    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"]
Oct 30 22:22:00.757958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:00.757926    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8daffd-phgzw"]
Oct 30 22:22:01.161775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:01.161734    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:01.791264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:01.791233    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f6c369-be9d-4cce-83ad-a8fe22e89a37" path="/var/lib/kubelet/pods/24f6c369-be9d-4cce-83ad-a8fe22e89a37/volumes"
Oct 30 22:22:04.080524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:04.080500    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:22:04.120847 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:04.120821    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:22:04.708113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:04.708079    5503 scope.go:122] "RemoveContainer" containerID="2d3c89194cc53faf7aa26df32cd70f3d53de2ff41afd63557758b1b30affb98c"
Oct 30 22:22:05.866456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:05.866422    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:07.195001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:07.194963    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:07.380839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:07.380805    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:07.563051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:07.562971    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:08.363573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.363538    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"]
Oct 30 22:22:08.364024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.364010    5503 cpu_manager.go:414] "containerMap: removing container" podUID="898932a1-57bb-4391-8c52-a6e5cde990d7" containerName="payload-processing"
Oct 30 22:22:08.364072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.364026    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898932a1-57bb-4391-8c52-a6e5cde990d7" containerName="payload-processing"
Oct 30 22:22:08.364072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.364055    5503 cpu_manager.go:414] "containerMap: removing container" podUID="24f6c369-be9d-4cce-83ad-a8fe22e89a37" containerName="revoke-keys"
Oct 30 22:22:08.364072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.364060    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f6c369-be9d-4cce-83ad-a8fe22e89a37" containerName="revoke-keys"
Oct 30 22:22:08.364186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.364140    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="898932a1-57bb-4391-8c52-a6e5cde990d7" containerName="payload-processing"
Oct 30 22:22:08.364186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.364152    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f6c369-be9d-4cce-83ad-a8fe22e89a37" containerName="revoke-keys"
Oct 30 22:22:08.368991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.368969    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.421239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.421201    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"]
Oct 30 22:22:08.446948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.446914    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2vn\" (UniqueName: \"kubernetes.io/projected/b230277f-9a9e-4b06-a576-5ba0dc627707-kube-api-access-ch2vn\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.446948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.446952    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.447165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.447136    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.469708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.469668    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:08.469708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.469696    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:22:08.469917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.469722    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:22:08.473007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.472988    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:22:08.548631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.548596    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.548856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.548760    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2vn\" (UniqueName: \"kubernetes.io/projected/b230277f-9a9e-4b06-a576-5ba0dc627707-kube-api-access-ch2vn\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.548856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.548782    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.549339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.549313    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.551025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.551006    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.611648 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.611593    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2vn\" (UniqueName: \"kubernetes.io/projected/b230277f-9a9e-4b06-a576-5ba0dc627707-kube-api-access-ch2vn\") pod \"maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:08.683316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:08.683288    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:09.053314 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:22:09.053270    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb230277f_9a9e_4b06_a576_5ba0dc627707.slice/crio-d341f91450e0e1f54b975aa11bcf891bae336507f477032b555cdf8212c04cb0 WatchSource:0}: Error finding container d341f91450e0e1f54b975aa11bcf891bae336507f477032b555cdf8212c04cb0: Status 404 returned error can't find the container with id d341f91450e0e1f54b975aa11bcf891bae336507f477032b555cdf8212c04cb0
Oct 30 22:22:09.056513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:09.056480    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"]
Oct 30 22:22:09.559696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:09.554406    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:09.865096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:09.865062    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb" event={"ID":"b230277f-9a9e-4b06-a576-5ba0dc627707","Type":"ContainerStarted","Data":"9e8dad31733071808d727fd09a7683ab2eda6221e47e1ba1a457db2e510be063"}
Oct 30 22:22:09.865309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:09.865121    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb" event={"ID":"b230277f-9a9e-4b06-a576-5ba0dc627707","Type":"ContainerStarted","Data":"d341f91450e0e1f54b975aa11bcf891bae336507f477032b555cdf8212c04cb0"}
Oct 30 22:22:09.986373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:09.986258    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb" podStartSLOduration=1.986244465 podStartE2EDuration="1.986244465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:09.984726826 +0000 UTC m=+2106.319303596" watchObservedRunningTime="2026-10-30 22:22:09.986244465 +0000 UTC m=+2106.320821234"
Oct 30 22:22:10.034726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:10.034690    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:10.694967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:10.694931    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"]
Oct 30 22:22:10.873381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:10.873349    5503 generic.go:356] "Generic (PLEG): container finished" podID="b230277f-9a9e-4b06-a576-5ba0dc627707" containerID="9e8dad31733071808d727fd09a7683ab2eda6221e47e1ba1a457db2e510be063" exitCode=0
Oct 30 22:22:10.873557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:10.873406    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb" event={"ID":"b230277f-9a9e-4b06-a576-5ba0dc627707","Type":"ContainerDied","Data":"9e8dad31733071808d727fd09a7683ab2eda6221e47e1ba1a457db2e510be063"}
Oct 30 22:22:10.949121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:10.949038    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:11.118126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:11.118087    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:11.292514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:11.292372    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:12.019860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.019832    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:12.127302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.127267    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b230277f-9a9e-4b06-a576-5ba0dc627707-kube-api-access-ch2vn\" (UniqueName: \"kubernetes.io/projected/b230277f-9a9e-4b06-a576-5ba0dc627707-kube-api-access-ch2vn\") pod \"b230277f-9a9e-4b06-a576-5ba0dc627707\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") "
Oct 30 22:22:12.127464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.127318    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-openshift-ca\") pod \"b230277f-9a9e-4b06-a576-5ba0dc627707\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") "
Oct 30 22:22:12.127464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.127424    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-xks-ca\") pod \"b230277f-9a9e-4b06-a576-5ba0dc627707\" (UID: \"b230277f-9a9e-4b06-a576-5ba0dc627707\") "
Oct 30 22:22:12.127718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.127695    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-openshift-ca" pod "b230277f-9a9e-4b06-a576-5ba0dc627707" (UID: "b230277f-9a9e-4b06-a576-5ba0dc627707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:12.127938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.127907    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:12.129367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.129341    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b230277f-9a9e-4b06-a576-5ba0dc627707-kube-api-access-ch2vn" pod "b230277f-9a9e-4b06-a576-5ba0dc627707" (UID: "b230277f-9a9e-4b06-a576-5ba0dc627707"). InnerVolumeSpecName "kube-api-access-ch2vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:12.129448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.129426    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-xks-ca" pod "b230277f-9a9e-4b06-a576-5ba0dc627707" (UID: "b230277f-9a9e-4b06-a576-5ba0dc627707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:12.251876 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.248582    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b230277f-9a9e-4b06-a576-5ba0dc627707-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:12.251876 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.248627    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch2vn\" (UniqueName: \"kubernetes.io/projected/b230277f-9a9e-4b06-a576-5ba0dc627707-kube-api-access-ch2vn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:12.730170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.730138    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:12.886520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.886495    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"
Oct 30 22:22:12.886736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.886530    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb" event={"ID":"b230277f-9a9e-4b06-a576-5ba0dc627707","Type":"ContainerDied","Data":"d341f91450e0e1f54b975aa11bcf891bae336507f477032b555cdf8212c04cb0"}
Oct 30 22:22:12.886736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:12.886578    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d341f91450e0e1f54b975aa11bcf891bae336507f477032b555cdf8212c04cb0"
Oct 30 22:22:15.695790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:15.695754    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t" podUID="085a50b4-b09e-4f90-b3c8-3b71217cf4ae" containerName="payload-processing" containerID="cri-o://17e7b889c14977835c53efb06d3e305e2b23e9094ffde04b0dac9bff27415ce2" gracePeriod=25
Oct 30 22:22:15.911557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:15.911509    5503 generic.go:356] "Generic (PLEG): container finished" podID="085a50b4-b09e-4f90-b3c8-3b71217cf4ae" containerID="17e7b889c14977835c53efb06d3e305e2b23e9094ffde04b0dac9bff27415ce2" exitCode=0
Oct 30 22:22:15.911778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:15.911621    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t" event={"ID":"085a50b4-b09e-4f90-b3c8-3b71217cf4ae","Type":"ContainerDied","Data":"17e7b889c14977835c53efb06d3e305e2b23e9094ffde04b0dac9bff27415ce2"}
Oct 30 22:22:15.982286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:15.982257    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:22:16.094896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.094866    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-kube-api-access-tlgkg\" (UniqueName: \"kubernetes.io/projected/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-kube-api-access-tlgkg\") pod \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") "
Oct 30 22:22:16.095092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.094914    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-plugins-config-volume\") pod \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\" (UID: \"085a50b4-b09e-4f90-b3c8-3b71217cf4ae\") "
Oct 30 22:22:16.095361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.095336    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-plugins-config-volume" pod "085a50b4-b09e-4f90-b3c8-3b71217cf4ae" (UID: "085a50b4-b09e-4f90-b3c8-3b71217cf4ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:16.095457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.095439    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:16.097096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.097075    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-kube-api-access-tlgkg" pod "085a50b4-b09e-4f90-b3c8-3b71217cf4ae" (UID: "085a50b4-b09e-4f90-b3c8-3b71217cf4ae"). InnerVolumeSpecName "kube-api-access-tlgkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:16.196123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.196086    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlgkg\" (UniqueName: \"kubernetes.io/projected/085a50b4-b09e-4f90-b3c8-3b71217cf4ae-kube-api-access-tlgkg\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:16.918066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.918032    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"
Oct 30 22:22:16.918066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.918063    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t" event={"ID":"085a50b4-b09e-4f90-b3c8-3b71217cf4ae","Type":"ContainerDied","Data":"452d48231c2a0fbfd16c2641a1c562c3fa44861255fce6a606e627eb5cab6949"}
Oct 30 22:22:16.918584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:16.918125    5503 scope.go:122] "RemoveContainer" containerID="17e7b889c14977835c53efb06d3e305e2b23e9094ffde04b0dac9bff27415ce2"
Oct 30 22:22:17.086321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:17.086288    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"]
Oct 30 22:22:17.090269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:17.090248    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a3357-6f547cbd69-g654t"]
Oct 30 22:22:17.372789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:17.372708    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:17.536917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:17.536881    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:17.742406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:17.742371    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:17.788995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:17.788956    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085a50b4-b09e-4f90-b3c8-3b71217cf4ae" path="/var/lib/kubelet/pods/085a50b4-b09e-4f90-b3c8-3b71217cf4ae/volumes"
Oct 30 22:22:21.579063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:21.579029    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:22:21.720239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:21.720200    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:22.110126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:22.110093    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:22.821864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:22.821834    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:26.624765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:26.624719    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:28.098694 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:22:28.098655    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:22:31.245726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:31.245689    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:35.933306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:35.933253    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:36.091256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:36.091193    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:37.288212 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.288182    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:37.737859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.737822    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"]
Oct 30 22:22:37.738315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.738299    5503 cpu_manager.go:414] "containerMap: removing container" podUID="085a50b4-b09e-4f90-b3c8-3b71217cf4ae" containerName="payload-processing"
Oct 30 22:22:37.738315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.738315    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085a50b4-b09e-4f90-b3c8-3b71217cf4ae" containerName="payload-processing"
Oct 30 22:22:37.738421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.738349    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b230277f-9a9e-4b06-a576-5ba0dc627707" containerName="revoke-keys"
Oct 30 22:22:37.738421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.738355    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b230277f-9a9e-4b06-a576-5ba0dc627707" containerName="revoke-keys"
Oct 30 22:22:37.738489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.738434    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b230277f-9a9e-4b06-a576-5ba0dc627707" containerName="revoke-keys"
Oct 30 22:22:37.738489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.738452    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="085a50b4-b09e-4f90-b3c8-3b71217cf4ae" containerName="payload-processing"
Oct 30 22:22:37.741990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.741966    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.803313 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.803281    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ebcc6d-openshift-default-dockercfg-9whs5\""
Oct 30 22:22:37.862056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.862014    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-certs\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.863112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.863083    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-credential-socket\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.863343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.863326    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-socket\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.863484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.863472    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-token\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.863651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.863621    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-envoy\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.864487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.864453    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:37.889741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.880721    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-data\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.890029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.890005    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nq2b\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-kube-api-access-2nq2b\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.890201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.890182    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istiod-ca-cert\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.890337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.890323    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-podinfo\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.994256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.994172    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"]
Oct 30 22:22:37.994973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.994907    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-credential-socket\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.994973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.994967    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-socket\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995001    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-token\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995050    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-envoy\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995102    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-data\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995128    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nq2b\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-kube-api-access-2nq2b\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995183    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istiod-ca-cert\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995234    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-podinfo\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995338    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-certs\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.995840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.995813    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-certs\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:37.999521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:37.999499    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istiod-ca-cert\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.001522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.001229    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-data\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.001522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.001477    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-credential-socket\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.001522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.001493    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-socket\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.022856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.020272    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-podinfo\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.022856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.022781    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-envoy\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.125657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.120576    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:38.127587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.120621    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-token\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.127921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.121304    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nq2b\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-kube-api-access-2nq2b\") pod \"e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") " pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.377822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.377731    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:38.814610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.814576    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"]
Oct 30 22:22:38.851978 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:22:38.851948    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1fcecc_4346_4ace_89d7_1cd3aecf541b.slice/crio-70edd46a408cf7f883228bca40a541839855733aff2fed1a2319ea4b2d7d945e WatchSource:0}: Error finding container 70edd46a408cf7f883228bca40a541839855733aff2fed1a2319ea4b2d7d945e: Status 404 returned error can't find the container with id 70edd46a408cf7f883228bca40a541839855733aff2fed1a2319ea4b2d7d945e
Oct 30 22:22:38.854458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.854422    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:22:38.854544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.854497    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:22:38.854544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:38.854528    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:22:39.054711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:39.054668    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" event={"ID":"3e1fcecc-4346-4ace-89d7-1cd3aecf541b","Type":"ContainerStarted","Data":"5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0"}
Oct 30 22:22:39.054915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:39.054729    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" event={"ID":"3e1fcecc-4346-4ace-89d7-1cd3aecf541b","Type":"ContainerStarted","Data":"70edd46a408cf7f883228bca40a541839855733aff2fed1a2319ea4b2d7d945e"}
Oct 30 22:22:39.178681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:39.178606    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podStartSLOduration=2.178591593 podStartE2EDuration="2.178591593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:39.176719692 +0000 UTC m=+2135.511296461" watchObservedRunningTime="2026-10-30 22:22:39.178591593 +0000 UTC m=+2135.513168362"
Oct 30 22:22:39.378551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:39.378508    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:39.380381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:39.380354    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:22:39.380528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:39.380443    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 22:22:40.383216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:40.383183    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:40.940330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:40.940292    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:41.067375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:41.067344    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:41.068405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:41.068383    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:22:41.987434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:41.987401    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:42.192519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:42.192489    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:42.374600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:42.374510    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:45.249708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:45.249668    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:45.439780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:45.439009    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:45.929016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:45.928972    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:46.558774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:46.558738    5503 patch_prober.go:29] interesting pod/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:46.559254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:46.558813    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:48.111619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:48.111582    5503 generic.go:356] "Generic (PLEG): container finished" podID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerID="c0d61e2b19da6d43e5555d3678631cbf9e39946d954034a83faca9ac18ab8245" exitCode=0
Oct 30 22:22:48.112079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:48.111678    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" event={"ID":"9ea557cd-e74e-4797-bdbc-bc338f21e2e1","Type":"ContainerDied","Data":"c0d61e2b19da6d43e5555d3678631cbf9e39946d954034a83faca9ac18ab8245"}
Oct 30 22:22:48.112123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:48.112087    5503 scope.go:122] "RemoveContainer" containerID="c0d61e2b19da6d43e5555d3678631cbf9e39946d954034a83faca9ac18ab8245"
Oct 30 22:22:48.113523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:48.113496    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:22:48.113577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:48.113562    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:22:48.113616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:48.113597    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:22:49.142007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:49.141964    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" event={"ID":"9ea557cd-e74e-4797-bdbc-bc338f21e2e1","Type":"ContainerStarted","Data":"b552ef0b0a57b3cd4caba5c648986d7d4b4eed4eeea65ebc22dc59658d086855"}
Oct 30 22:22:49.324084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:49.324046    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:22:49.324440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:49.324417    5503 patch_prober.go:29] interesting pod/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:49.328656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:49.324590    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:22:50.337669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:50.336821    5503 patch_prober.go:29] interesting pod/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:50.337669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:50.336884    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:22:50.868858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:50.868812    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:51.339107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:51.339069    5503 patch_prober.go:29] interesting pod/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:51.339744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:51.339713    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:22:52.338787 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:52.338746    5503 patch_prober.go:29] interesting pod/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:52.339088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:52.339054    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:22:52.580255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:52.580214    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:22:52.641154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:52.641032    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:53.327068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:53.327031    5503 patch_prober.go:29] interesting pod/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:53.327358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:53.327328    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:22:54.056482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:54.056446    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"]
Oct 30 22:22:54.104734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:54.104704    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e30260-m2gnx"]
Oct 30 22:22:54.323717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:54.322066    5503 patch_prober.go:29] interesting pod/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:22:54.323717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:54.322127    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:22:55.325974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:55.325935    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:22:55.326431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:55.326226    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:22:55.326846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:55.326827    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:22:55.580234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:55.580149    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:55.795063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:55.795030    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce98b14f-e1b5-4885-b656-2262f87c17f5" path="/var/lib/kubelet/pods/ce98b14f-e1b5-4885-b656-2262f87c17f5/volumes"
Oct 30 22:22:56.716051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:56.716015    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:57.334368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:57.334332    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:57.514425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:22:57.514390    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:22:58.099715 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:22:58.099678    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:23:01.912481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:01.912444    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.088742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:02.088706    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.248010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:02.247932    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:02.556329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:02.556254    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:04.768539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:04.768508    5503 scope.go:122] "RemoveContainer" containerID="6e409f10c09f2ed1f933fb25969b80b1355f2fc48826405f062fd39d117ca2b0"
Oct 30 22:23:04.779162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:04.779146    5503 scope.go:122] "RemoveContainer" containerID="08fcd5590ef4fe32053365a470a606445f2086bc7299f27484d4d6d84c39ba2d"
Oct 30 22:23:05.180748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:05.180708    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:05.365622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:05.365580    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:05.910504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:05.910469    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"]
Oct 30 22:23:05.915769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:05.915748    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:05.924922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:05.924894    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c7c164-2675-4180-b20e-87aa65686a4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:05.925037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:05.925022    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/d5c7c164-2675-4180-b20e-87aa65686a4d-kube-api-access-h5zm7\") pod \"payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:05.968906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:05.968875    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"]
Oct 30 22:23:06.017926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.017892    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-923d78-dockercfg-7rr66\""
Oct 30 22:23:06.017926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.017892    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-923d78\""
Oct 30 22:23:06.026145 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.026117    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c7c164-2675-4180-b20e-87aa65686a4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:06.026237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.026189    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/d5c7c164-2675-4180-b20e-87aa65686a4d-kube-api-access-h5zm7\") pod \"payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:06.026786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.026770    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c7c164-2675-4180-b20e-87aa65686a4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:06.088693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.088661    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/d5c7c164-2675-4180-b20e-87aa65686a4d-kube-api-access-h5zm7\") pod \"payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:06.134834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.134799    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:06.228563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.228477    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:06.609917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.609894    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"]
Oct 30 22:23:06.610781 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:23:06.610751    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c7c164_2675_4180_b20e_87aa65686a4d.slice/crio-dbcabd0bd5e0b116d08ccd6b2c32bb33be193c3cafa5eba92d10e55db6124336 WatchSource:0}: Error finding container dbcabd0bd5e0b116d08ccd6b2c32bb33be193c3cafa5eba92d10e55db6124336: Status 404 returned error can't find the container with id dbcabd0bd5e0b116d08ccd6b2c32bb33be193c3cafa5eba92d10e55db6124336
Oct 30 22:23:06.646319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:06.646293    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:07.253309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:07.253263    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267" event={"ID":"d5c7c164-2675-4180-b20e-87aa65686a4d","Type":"ContainerStarted","Data":"6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912"}
Oct 30 22:23:07.253852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:07.253335    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267" event={"ID":"d5c7c164-2675-4180-b20e-87aa65686a4d","Type":"ContainerStarted","Data":"dbcabd0bd5e0b116d08ccd6b2c32bb33be193c3cafa5eba92d10e55db6124336"}
Oct 30 22:23:07.253852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:07.253435    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:07.389777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:07.389722    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267" podStartSLOduration=2.389706827 podStartE2EDuration="2.389706827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:07.387365841 +0000 UTC m=+2163.721942609" watchObservedRunningTime="2026-10-30 22:23:07.389706827 +0000 UTC m=+2163.724283611"
Oct 30 22:23:08.677192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:08.677155    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:09.225934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:09.225898    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:09.399200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:09.399164    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.400653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:10.400606    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.684250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:10.684174    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.833763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:10.833728    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:10.875733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:10.875705    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"]
Oct 30 22:23:10.926155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:10.926124    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7350bc-v2krb"]
Oct 30 22:23:11.322922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:11.322884    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:11.788831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:11.788799    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b230277f-9a9e-4b06-a576-5ba0dc627707" path="/var/lib/kubelet/pods/b230277f-9a9e-4b06-a576-5ba0dc627707/volumes"
Oct 30 22:23:15.422748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:15.422666    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:15.664940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:15.664906    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:17.894370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:17.894336    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:18.049566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:18.049535    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:18.228762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:18.228730    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:18.265906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:18.265882    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:23:19.798805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:19.798769    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.336978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:20.336943    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.570235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:20.570201    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:20.786744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:20.786704    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:22.081238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:22.081189    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:23.159752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:23.159720    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:25.127281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:25.127247    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:25.283109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:25.283074    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:25.435607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:25.435575    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:25.817999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:25.817902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:26.743707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:26.743674    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.438816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:27.438768    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.640356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:27.640320    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:27.814252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:27.814163    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:28.104163 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:23:28.104070    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:23:30.248991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:30.248957    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:31.379068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.379031    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:31.568163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.568072    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:31.802301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.802267    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"]
Oct 30 22:23:31.805723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.805698    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:31.859627 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.859599    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ae8879\""
Oct 30 22:23:31.917374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.917337    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ae8879-dockercfg-5jsxr\""
Oct 30 22:23:31.941660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.940320    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb2g\" (UniqueName: \"kubernetes.io/projected/5270aacc-de51-4e7c-9be2-f468c64cbf23-kube-api-access-2sb2g\") pod \"payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:31.941660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.940543    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5270aacc-de51-4e7c-9be2-f468c64cbf23-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:31.944009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.943982    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"]
Oct 30 22:23:31.971694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.967428    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"]
Oct 30 22:23:31.989483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:31.989453    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.002902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.002877    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"]
Oct 30 22:23:32.041396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.041362    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5270aacc-de51-4e7c-9be2-f468c64cbf23-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:32.041592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.041496    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb2g\" (UniqueName: \"kubernetes.io/projected/5270aacc-de51-4e7c-9be2-f468c64cbf23-kube-api-access-2sb2g\") pod \"payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:32.042179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.042160    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5270aacc-de51-4e7c-9be2-f468c64cbf23-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:32.106773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.106698    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb2g\" (UniqueName: \"kubernetes.io/projected/5270aacc-de51-4e7c-9be2-f468c64cbf23-kube-api-access-2sb2g\") pod \"payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:32.120605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.120577    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:32.127270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.127243    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"]
Oct 30 22:23:32.136041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.136012    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.142643 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.142607    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f508d76d-143e-4498-af63-9eee39e7336c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.143026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.142997    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxhf\" (UniqueName: \"kubernetes.io/projected/f508d76d-143e-4498-af63-9eee39e7336c-kube-api-access-htxhf\") pod \"payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.184926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.184859    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"]
Oct 30 22:23:32.191307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.191149    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0e90fe\"/\"openshift-service-ca.crt\""
Oct 30 22:23:32.191307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.191205    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0e90fe\"/\"kube-root-ca.crt\""
Oct 30 22:23:32.191518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.191466    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0e90fe\"/\"ipp-route-0e90fe-kserve-self-signed-certs\""
Oct 30 22:23:32.191730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.191680    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0e90fe\"/\"default-dockercfg-ksj5f\""
Oct 30 22:23:32.223862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.223827    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:32.244280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244242    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f508d76d-143e-4498-af63-9eee39e7336c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.244482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244377    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-home\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.244482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244408    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-model-cache\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.244617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244484    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-dshm\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.244617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244533    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1024f0de-a385-4afd-aed8-91259a20574f-tls-certs\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.244617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244556    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-tmp-dir\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.244807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244668    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxd7m\" (UniqueName: \"kubernetes.io/projected/1024f0de-a385-4afd-aed8-91259a20574f-kube-api-access-vxd7m\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.244807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.244756    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxhf\" (UniqueName: \"kubernetes.io/projected/f508d76d-143e-4498-af63-9eee39e7336c-kube-api-access-htxhf\") pod \"payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.245151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.245126    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f508d76d-143e-4498-af63-9eee39e7336c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.334389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.332494    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxhf\" (UniqueName: \"kubernetes.io/projected/f508d76d-143e-4498-af63-9eee39e7336c-kube-api-access-htxhf\") pod \"payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.345903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.345864    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-home\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.345918    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-model-cache\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.345956    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-dshm\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.345971    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1024f0de-a385-4afd-aed8-91259a20574f-tls-certs\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.345993    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-tmp-dir\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.346045    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxd7m\" (UniqueName: \"kubernetes.io/projected/1024f0de-a385-4afd-aed8-91259a20574f-kube-api-access-vxd7m\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.346288    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-home\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.346428    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-tmp-dir\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.346665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.346648    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-model-cache\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.348178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.348160    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-dshm\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.348377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.348360    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1024f0de-a385-4afd-aed8-91259a20574f-tls-certs\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.408858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.408821    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxd7m\" (UniqueName: \"kubernetes.io/projected/1024f0de-a385-4afd-aed8-91259a20574f-kube-api-access-vxd7m\") pod \"ipp-route-0e90fe-kserve-64c675dbfc-tmhdm\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") " pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.475076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.475035    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:32.512930 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:23:32.512894    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5270aacc_de51_4e7c_9be2_f468c64cbf23.slice/crio-c848fe9c3863481c0bacf01c7ba15bdbb89aeeac192719dd4cad58ccf841add4 WatchSource:0}: Error finding container c848fe9c3863481c0bacf01c7ba15bdbb89aeeac192719dd4cad58ccf841add4: Status 404 returned error can't find the container with id c848fe9c3863481c0bacf01c7ba15bdbb89aeeac192719dd4cad58ccf841add4
Oct 30 22:23:32.515072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.515047    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"]
Oct 30 22:23:32.612271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.612223    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:32.860436 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:23:32.860407    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1024f0de_a385_4afd_aed8_91259a20574f.slice/crio-f90880d82df5d7878158a9eeec20e060781a930eec75c633da66ae7083026fd9 WatchSource:0}: Error finding container f90880d82df5d7878158a9eeec20e060781a930eec75c633da66ae7083026fd9: Status 404 returned error can't find the container with id f90880d82df5d7878158a9eeec20e060781a930eec75c633da66ae7083026fd9
Oct 30 22:23:32.863148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:32.863127    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"]
Oct 30 22:23:33.092291 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:23:33.092259    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf508d76d_143e_4498_af63_9eee39e7336c.slice/crio-3513a08abcfb82ed16a6aaf80a194e6e8ea9287a8d5cd0b03605b600d2d789a4 WatchSource:0}: Error finding container 3513a08abcfb82ed16a6aaf80a194e6e8ea9287a8d5cd0b03605b600d2d789a4: Status 404 returned error can't find the container with id 3513a08abcfb82ed16a6aaf80a194e6e8ea9287a8d5cd0b03605b600d2d789a4
Oct 30 22:23:33.094527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.094507    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"]
Oct 30 22:23:33.492278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.492238    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm" event={"ID":"1024f0de-a385-4afd-aed8-91259a20574f","Type":"ContainerStarted","Data":"beb198bf2bc1c57bcf3a2f5f2e5ec924ae182923cd6616f92f90595e04920869"}
Oct 30 22:23:33.492760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.492297    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm" event={"ID":"1024f0de-a385-4afd-aed8-91259a20574f","Type":"ContainerStarted","Data":"f90880d82df5d7878158a9eeec20e060781a930eec75c633da66ae7083026fd9"}
Oct 30 22:23:33.493797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.493772    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs" event={"ID":"f508d76d-143e-4498-af63-9eee39e7336c","Type":"ContainerStarted","Data":"454c273d83431728e319475d01c228d52848e646d941dfea17cb65dd0102e183"}
Oct 30 22:23:33.493897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.493821    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs" event={"ID":"f508d76d-143e-4498-af63-9eee39e7336c","Type":"ContainerStarted","Data":"3513a08abcfb82ed16a6aaf80a194e6e8ea9287a8d5cd0b03605b600d2d789a4"}
Oct 30 22:23:33.493897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.493858    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:33.495366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.495344    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp" event={"ID":"5270aacc-de51-4e7c-9be2-f468c64cbf23","Type":"ContainerStarted","Data":"fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630"}
Oct 30 22:23:33.495464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.495383    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp" event={"ID":"5270aacc-de51-4e7c-9be2-f468c64cbf23","Type":"ContainerStarted","Data":"c848fe9c3863481c0bacf01c7ba15bdbb89aeeac192719dd4cad58ccf841add4"}
Oct 30 22:23:33.495464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.495403    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:33.537456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.537426    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:33.621048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.620999    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm" podStartSLOduration=1.620985605 podStartE2EDuration="1.620985605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:33.619195034 +0000 UTC m=+2189.953771803" watchObservedRunningTime="2026-10-30 22:23:33.620985605 +0000 UTC m=+2189.955562373"
Oct 30 22:23:33.691773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.691738    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:33.751547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.751401    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs" podStartSLOduration=2.751382838 podStartE2EDuration="2.751382838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:33.74992295 +0000 UTC m=+2190.084499721" watchObservedRunningTime="2026-10-30 22:23:33.751382838 +0000 UTC m=+2190.085959601"
Oct 30 22:23:33.897424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:33.897349    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp" podStartSLOduration=2.897324969 podStartE2EDuration="2.897324969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:33.87403717 +0000 UTC m=+2190.208613951" watchObservedRunningTime="2026-10-30 22:23:33.897324969 +0000 UTC m=+2190.231901738"
Oct 30 22:23:34.475906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:34.475878    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:35.248689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:35.248653    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:38.959855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:38.959802    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:39.154814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:39.154781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:40.086738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:40.086701    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:40.294172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:40.290368    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:40.456565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:40.456511    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:41.618952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:41.618920    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:23:42.476072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:42.476035    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:42.489544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:42.489514    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:42.600065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:42.600033    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:23:44.504327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:44.504301    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:23:44.504739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:44.504362    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:23:45.225319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:45.225283    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:49.717052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:49.717017    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:54.356819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:23:54.356783    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:23:58.098301 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:23:58.098270    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:24:00.508202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:00.508139    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:04.815832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:04.815793    5503 scope.go:122] "RemoveContainer" containerID="ce98d68d2da2c793f13f0faafbdccfafb9259c1976fc1bc439bf59ec87ea5ef4"
Oct 30 22:24:04.928178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:04.928144    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:05.161784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:05.161747    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:05.322917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:05.322880    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:09.830110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:09.830058    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:16.009131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:16.009091    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:17.904607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:17.904574    5503 generic.go:356] "Generic (PLEG): container finished" podID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerID="5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0" exitCode=0
Oct 30 22:24:17.905177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:17.904648    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" event={"ID":"3e1fcecc-4346-4ace-89d7-1cd3aecf541b","Type":"ContainerDied","Data":"5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0"}
Oct 30 22:24:17.905177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:17.905129    5503 scope.go:122] "RemoveContainer" containerID="5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0"
Oct 30 22:24:17.906724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:17.906690    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:17.906820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:17.906768    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:17.906820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:17.906802    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:18.912759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:18.912718    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" event={"ID":"3e1fcecc-4346-4ace-89d7-1cd3aecf541b","Type":"ContainerStarted","Data":"a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5"}
Oct 30 22:24:19.378209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:19.378128    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:24:19.378366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:19.378308    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:24:19.378366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:19.378354    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 22:24:20.378229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:20.378192    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:24:20.378666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:20.378250    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 22:24:20.755338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:20.755301    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:21.379127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:21.379089    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:24:21.379543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:21.379156    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 22:24:22.378625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:22.378590    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:24:22.378841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:22.378683    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 22:24:22.397596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:22.397558    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:23.378862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:23.378823    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:24:23.379046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:23.378881    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 22:24:23.531662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:23.530434    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:24.207454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:24.207414    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:24.389660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:24.383007    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 30 22:24:24.389660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:24.383067    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 30 22:24:24.462164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:24.462079    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:24:24.616362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:24.616331    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:24.791552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:24.791468    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:25.382492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:25.382459    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:24:25.383017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:25.382990    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:24:25.383739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:25.383721    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:24:27.162785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:27.162748    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:28.113822 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:24:28.113787    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:24:31.685726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:31.685691    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:31.883498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:31.883458    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:32.064795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:32.064711    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:32.934023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:32.933989    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:35.050554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:35.050522    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"]
Oct 30 22:24:35.103480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:35.103440    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686cb19f-km9m4"]
Oct 30 22:24:35.788894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:35.788859    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2" path="/var/lib/kubelet/pods/3db7bdbe-c32e-4cd7-bcb7-d93da4564fe2/volumes"
Oct 30 22:24:36.438661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:36.438599    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:39.766736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:39.766702    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"]
Oct 30 22:24:39.767204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:39.766942    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" containerID="cri-o://348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7" gracePeriod=30
Oct 30 22:24:41.014904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:41.014868    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:41.216561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:41.216523    5503 patch_prober.go:29] interesting pod/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:24:41.216747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:41.216587    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:24:45.041409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.041376    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:24:45.101676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.101585    5503 generic.go:356] "Generic (PLEG): container finished" podID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerID="348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7" exitCode=0
Oct 30 22:24:45.101676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.101650    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"
Oct 30 22:24:45.101836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.101672    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" event={"ID":"99e6b256-5f2e-4dc1-96ca-001bb509a8ad","Type":"ContainerDied","Data":"348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7"}
Oct 30 22:24:45.101836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.101759    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj" event={"ID":"99e6b256-5f2e-4dc1-96ca-001bb509a8ad","Type":"ContainerDied","Data":"a3afafe7e2dfb8795c2fbec34e30427833384da6cd0ef22667df07248278c441"}
Oct 30 22:24:45.101836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.101789    5503 scope.go:122] "RemoveContainer" containerID="348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7"
Oct 30 22:24:45.111406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.111387    5503 scope.go:122] "RemoveContainer" containerID="348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7"
Oct 30 22:24:45.111701 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:24:45.111678    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7\": container with ID starting with 348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7 not found: ID does not exist" containerID="348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7"
Oct 30 22:24:45.111798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.111708    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7"} err="failed to get container status \"348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7\": rpc error: code = NotFound desc = could not find container \"348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7\": container with ID starting with 348f98def5e2ecf24c84cc3e47c83bc5f6d1ef906a71e165eb43c4798ade54a7 not found: ID does not exist"
Oct 30 22:24:45.220444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220414    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-podinfo\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.220604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220455    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-credential-socket\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.220604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220491    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-certs\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.220741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220620    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-kube-api-access-k76ts\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-kube-api-access-k76ts\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.220741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220716    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-data\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.220853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220777    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-envoy\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.220853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220806    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-certs" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:45.220853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220832    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istiod-ca-cert\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.221014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220872    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-token\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-token\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.221014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.220816    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-credential-socket" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:45.221111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221019    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-socket\") pod \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\" (UID: \"99e6b256-5f2e-4dc1-96ca-001bb509a8ad\") "
Oct 30 22:24:45.221111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221028    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-data" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:45.221257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221235    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-socket" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:45.221565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221540    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.221716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221572    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.221716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221590    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.221716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221605    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.221861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.221752    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istiod-ca-cert" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:45.222809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.222781    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-podinfo" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:45.222942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.222922    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-envoy" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:45.223329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.223309    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-token" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:45.223389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.223330    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-kube-api-access-k76ts" pod "99e6b256-5f2e-4dc1-96ca-001bb509a8ad" (UID: "99e6b256-5f2e-4dc1-96ca-001bb509a8ad"). InnerVolumeSpecName "kube-api-access-k76ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:45.322724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.322678    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.322955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.322933    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.323302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.322957    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.323302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.322975    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.323302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.322990    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76ts\" (UniqueName: \"kubernetes.io/projected/99e6b256-5f2e-4dc1-96ca-001bb509a8ad-kube-api-access-k76ts\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:45.422586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.422555    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.526038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.526006    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"]
Oct 30 22:24:45.529751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.529729    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d09fff-openshift-default-86957cb9c9-kl5gj"]
Oct 30 22:24:45.572902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.572864    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.726367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.726271    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:45.789540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:45.789508    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" path="/var/lib/kubelet/pods/99e6b256-5f2e-4dc1-96ca-001bb509a8ad/volumes"
Oct 30 22:24:50.217766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:50.217731    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:51.262585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.262551    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"]
Oct 30 22:24:51.263052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.263037    5503 cpu_manager.go:414] "containerMap: removing container" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy"
Oct 30 22:24:51.263105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.263054    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy"
Oct 30 22:24:51.263179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.263169    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e6b256-5f2e-4dc1-96ca-001bb509a8ad" containerName="istio-proxy"
Oct 30 22:24:51.267698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.267674    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301315    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301378    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301471    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c297409-3d79-40ba-b6c3-d883887f7933-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301575    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47kzl\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-kube-api-access-47kzl\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301607    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c297409-3d79-40ba-b6c3-d883887f7933-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301628    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301812    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301844    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.301898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.301878    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.314005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.313981    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"]
Oct 30 22:24:51.317784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.317765    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.327113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.327093    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ae8879-openshift-default-dockercfg-j62zp\""
Oct 30 22:24:51.365214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.365187    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"]
Oct 30 22:24:51.403397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403252    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.403614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403586    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.403769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403667    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.403769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403755    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.403906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403798    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.403906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403821    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11341b43-f930-4bdd-8a93-0c4ed09e484c-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.403906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403858    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.403906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403895    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rh6\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-kube-api-access-q6rh6\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.404121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403921    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c297409-3d79-40ba-b6c3-d883887f7933-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403982    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.404121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403999    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404015    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47kzl\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-kube-api-access-47kzl\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404058    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c297409-3d79-40ba-b6c3-d883887f7933-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404090    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404166    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.404413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404195    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.404413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.403751    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404247    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404370    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404431    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.404679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404534    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.404935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.404907    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.405354 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.405331    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c297409-3d79-40ba-b6c3-d883887f7933-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.406880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.406853    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c297409-3d79-40ba-b6c3-d883887f7933-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.407117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.407050    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.410497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.410447    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"]
Oct 30 22:24:51.465975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.465944    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47kzl\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-kube-api-access-47kzl\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.466220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.466201    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.505854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.505622    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506123    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506163    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506205    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506248    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506305    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506337    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506360    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11341b43-f930-4bdd-8a93-0c4ed09e484c-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506432    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rh6\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-kube-api-access-q6rh6\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506586    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-credential-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506750    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-certs\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.506932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.506910    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-socket\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.507074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.507048    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-data\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.507246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.507227    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11341b43-f930-4bdd-8a93-0c4ed09e484c-istiod-ca-cert\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.508647 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.508619    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-envoy\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.509331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.509307    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-podinfo\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.568099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.568027    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-token\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.568471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.568450    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rh6\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-kube-api-access-q6rh6\") pod \"e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") " pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:51.584373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.584344    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:51.632086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:51.632051    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:52.010458 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:24:52.010415    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c297409_3d79_40ba_b6c3_d883887f7933.slice/crio-e1ce1d8fbb6c2814bc7b374ab56281a584d94df84c053d7544017a11f6a7b5b5 WatchSource:0}: Error finding container e1ce1d8fbb6c2814bc7b374ab56281a584d94df84c053d7544017a11f6a7b5b5: Status 404 returned error can't find the container with id e1ce1d8fbb6c2814bc7b374ab56281a584d94df84c053d7544017a11f6a7b5b5
Oct 30 22:24:52.014756 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.014719    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"]
Oct 30 22:24:52.018893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.018564    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:52.018893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.018731    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:52.018893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.018816    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:52.113240 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:24:52.113209    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11341b43_f930_4bdd_8a93_0c4ed09e484c.slice/crio-2dfca6d29b320ab352e50ed66de84d681f241977363ebfe34726324ff6d7a0d7 WatchSource:0}: Error finding container 2dfca6d29b320ab352e50ed66de84d681f241977363ebfe34726324ff6d7a0d7: Status 404 returned error can't find the container with id 2dfca6d29b320ab352e50ed66de84d681f241977363ebfe34726324ff6d7a0d7
Oct 30 22:24:52.115803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.115775    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"]
Oct 30 22:24:52.116936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.116889    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:52.117016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.116996    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:52.117062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.117050    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:24:52.154745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.154706    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" event={"ID":"2c297409-3d79-40ba-b6c3-d883887f7933","Type":"ContainerStarted","Data":"09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573"}
Oct 30 22:24:52.154899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.154778    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" event={"ID":"2c297409-3d79-40ba-b6c3-d883887f7933","Type":"ContainerStarted","Data":"e1ce1d8fbb6c2814bc7b374ab56281a584d94df84c053d7544017a11f6a7b5b5"}
Oct 30 22:24:52.156757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.156725    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" event={"ID":"11341b43-f930-4bdd-8a93-0c4ed09e484c","Type":"ContainerStarted","Data":"2dfca6d29b320ab352e50ed66de84d681f241977363ebfe34726324ff6d7a0d7"}
Oct 30 22:24:52.280928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.280809    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" podStartSLOduration=1.280790208 podStartE2EDuration="1.280790208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:52.278204636 +0000 UTC m=+2268.612781406" watchObservedRunningTime="2026-10-30 22:24:52.280790208 +0000 UTC m=+2268.615366978"
Oct 30 22:24:52.584947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:52.584857    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:53.173990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:53.173947    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" event={"ID":"11341b43-f930-4bdd-8a93-0c4ed09e484c","Type":"ContainerStarted","Data":"1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee"}
Oct 30 22:24:53.303199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:53.303138    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" podStartSLOduration=2.303125139 podStartE2EDuration="2.303125139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:53.301266815 +0000 UTC m=+2269.635843573" watchObservedRunningTime="2026-10-30 22:24:53.303125139 +0000 UTC m=+2269.637701907"
Oct 30 22:24:53.586909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:53.586821    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:24:53.586909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:53.586874    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 22:24:53.649449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:53.649415    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:53.661659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:53.651126    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:24:53.661659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:53.651183    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 22:24:54.584811 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:54.584774    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:24:54.585211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:54.584831    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 22:24:54.633009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:54.632973    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:24:54.633279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:54.633242    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 22:24:54.808582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:54.808540    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:24:55.353607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.353566    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"]
Oct 30 22:24:55.367703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.367340    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.381924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.381894    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rv6h\" (UniqueName: \"kubernetes.io/projected/51848ac5-bd14-4858-bbe9-756075fac717-kube-api-access-2rv6h\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.382091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.382009    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.382158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.382142    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.395856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.395829    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"]
Oct 30 22:24:55.458200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.458157    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:24:55.458543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.458525    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:55.458808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.458776    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:24:55.472266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.472237    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:24:55.482911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.482883    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.483015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.482981    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.483015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.483008    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rv6h\" (UniqueName: \"kubernetes.io/projected/51848ac5-bd14-4858-bbe9-756075fac717-kube-api-access-2rv6h\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.483663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.483625    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.485338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.485306    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.568501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.568445    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rv6h\" (UniqueName: \"kubernetes.io/projected/51848ac5-bd14-4858-bbe9-756075fac717-kube-api-access-2rv6h\") pod \"maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:55.594678 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.594624    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:24:55.595129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.594692    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 22:24:55.636139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.636044    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:24:55.637507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.637473    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 22:24:55.685456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:55.685427    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:56.167665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:56.166172    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"]
Oct 30 22:24:56.200528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:56.200503    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28" event={"ID":"51848ac5-bd14-4858-bbe9-756075fac717","Type":"ContainerStarted","Data":"5bb805576b701709da9835ae07246b5fb0c5770c1f98e4ef12119ddf0d00a054"}
Oct 30 22:24:56.584679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:56.584616    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:24:56.584870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:56.584698    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 22:24:56.637666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:56.634943    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:24:56.637666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:56.635000    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 22:24:57.208001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:57.207968    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28" event={"ID":"51848ac5-bd14-4858-bbe9-756075fac717","Type":"ContainerStarted","Data":"626bd6e815cd30b7f4c985253cc8d777bf9da2311ed09d32439a24d2b85cdcb8"}
Oct 30 22:24:57.332086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:57.332023    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28" podStartSLOduration=2.332003111 podStartE2EDuration="2.332003111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:57.330513515 +0000 UTC m=+2273.665090284" watchObservedRunningTime="2026-10-30 22:24:57.332003111 +0000 UTC m=+2273.666579882"
Oct 30 22:24:57.585073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:57.584990    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:24:57.585073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:57.585044    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Oct 30 22:24:57.633659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:57.633052    5503 patch_prober.go:29] interesting pod/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Oct 30 22:24:57.633659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:57.633113    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Oct 30 22:24:58.100034 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:24:58.100001    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:24:58.218682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.216118    5503 generic.go:356] "Generic (PLEG): container finished" podID="51848ac5-bd14-4858-bbe9-756075fac717" containerID="626bd6e815cd30b7f4c985253cc8d777bf9da2311ed09d32439a24d2b85cdcb8" exitCode=0
Oct 30 22:24:58.218682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.216328    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28" event={"ID":"51848ac5-bd14-4858-bbe9-756075fac717","Type":"ContainerDied","Data":"626bd6e815cd30b7f4c985253cc8d777bf9da2311ed09d32439a24d2b85cdcb8"}
Oct 30 22:24:58.589717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.589687    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:58.590112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.590092    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:58.590781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.590759    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:24:58.637784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.637717    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:58.637995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.637978    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:58.638502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:58.638483    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:24:59.383666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.383631    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:24:59.437215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.437185    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-xks-ca\") pod \"51848ac5-bd14-4858-bbe9-756075fac717\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") "
Oct 30 22:24:59.437362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.437278    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51848ac5-bd14-4858-bbe9-756075fac717-kube-api-access-2rv6h\" (UniqueName: \"kubernetes.io/projected/51848ac5-bd14-4858-bbe9-756075fac717-kube-api-access-2rv6h\") pod \"51848ac5-bd14-4858-bbe9-756075fac717\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") "
Oct 30 22:24:59.437471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.437453    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-openshift-ca\") pod \"51848ac5-bd14-4858-bbe9-756075fac717\" (UID: \"51848ac5-bd14-4858-bbe9-756075fac717\") "
Oct 30 22:24:59.437823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.437799    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-openshift-ca" pod "51848ac5-bd14-4858-bbe9-756075fac717" (UID: "51848ac5-bd14-4858-bbe9-756075fac717"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:59.439237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.439216    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-xks-ca" pod "51848ac5-bd14-4858-bbe9-756075fac717" (UID: "51848ac5-bd14-4858-bbe9-756075fac717"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:59.439329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.439263    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51848ac5-bd14-4858-bbe9-756075fac717-kube-api-access-2rv6h" pod "51848ac5-bd14-4858-bbe9-756075fac717" (UID: "51848ac5-bd14-4858-bbe9-756075fac717"). InnerVolumeSpecName "kube-api-access-2rv6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:59.538508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.538431    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.538508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.538457    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rv6h\" (UniqueName: \"kubernetes.io/projected/51848ac5-bd14-4858-bbe9-756075fac717-kube-api-access-2rv6h\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.538508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.538467    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51848ac5-bd14-4858-bbe9-756075fac717-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:59.841056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:24:59.840953    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:00.234621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:00.234592    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"
Oct 30 22:25:00.234804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:00.234626    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28" event={"ID":"51848ac5-bd14-4858-bbe9-756075fac717","Type":"ContainerDied","Data":"5bb805576b701709da9835ae07246b5fb0c5770c1f98e4ef12119ddf0d00a054"}
Oct 30 22:25:00.234804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:00.234701    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bb805576b701709da9835ae07246b5fb0c5770c1f98e4ef12119ddf0d00a054"
Oct 30 22:25:04.623446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:04.623415    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:04.840428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:04.840396    5503 scope.go:122] "RemoveContainer" containerID="9e732728939ba0df28ce883f2ed6291d6727e9474356c7175ecc24028d101666"
Oct 30 22:25:05.470871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:05.470826    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:08.644203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.644173    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"]
Oct 30 22:25:08.644707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.644691    5503 cpu_manager.go:414] "containerMap: removing container" podUID="51848ac5-bd14-4858-bbe9-756075fac717" containerName="revoke-keys"
Oct 30 22:25:08.644761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.644709    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51848ac5-bd14-4858-bbe9-756075fac717" containerName="revoke-keys"
Oct 30 22:25:08.644833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.644824    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="51848ac5-bd14-4858-bbe9-756075fac717" containerName="revoke-keys"
Oct 30 22:25:08.649284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.649257    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.717216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.717181    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"]
Oct 30 22:25:08.740235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.740208    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.740404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.740257    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9pf6\" (UniqueName: \"kubernetes.io/projected/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-kube-api-access-w9pf6\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.740472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.740419    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.765681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.765631    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:08.765854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.765787    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:25:08.765854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.765789    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:08.765854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.765816    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:08.841590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.841556    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9pf6\" (UniqueName: \"kubernetes.io/projected/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-kube-api-access-w9pf6\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.841807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.841647    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.841885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.841806    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.842526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.842499    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.844288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.844263    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.904523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.904493    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9pf6\" (UniqueName: \"kubernetes.io/projected/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-kube-api-access-w9pf6\") pod \"maas-api-revoke-sub-6eb5a856-l9lsn\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:08.962837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:08.962794    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:09.334190 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:25:09.334162    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9f5047_0b99_4ddc_9482_3cfdcb2b26df.slice/crio-27e65ec19ef7681cb8fa7176a99f7fed1d528df939626c840f4a0ec6041865d7 WatchSource:0}: Error finding container 27e65ec19ef7681cb8fa7176a99f7fed1d528df939626c840f4a0ec6041865d7: Status 404 returned error can't find the container with id 27e65ec19ef7681cb8fa7176a99f7fed1d528df939626c840f4a0ec6041865d7
Oct 30 22:25:09.337202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:09.337180    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"]
Oct 30 22:25:09.352834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:09.352799    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:10.296320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.296293    5503 generic.go:356] "Generic (PLEG): container finished" podID="dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" containerID="72a2c8863e21ad78d6894fe3d8fa1463ac702074ecc98005c514c59adb272024" exitCode=0
Oct 30 22:25:10.296672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.296365    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn" event={"ID":"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df","Type":"ContainerDied","Data":"72a2c8863e21ad78d6894fe3d8fa1463ac702074ecc98005c514c59adb272024"}
Oct 30 22:25:10.296672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.296421    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn" event={"ID":"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df","Type":"ContainerStarted","Data":"27e65ec19ef7681cb8fa7176a99f7fed1d528df939626c840f4a0ec6041865d7"}
Oct 30 22:25:10.415663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.415618    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"]
Oct 30 22:25:10.419753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.419732    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.461253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.461218    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqb5\" (UniqueName: \"kubernetes.io/projected/3200469f-32a5-4998-b373-23fc735afe4b-kube-api-access-lbqb5\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.461399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.461363    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.461459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.461405    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.475076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.475043    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"]
Oct 30 22:25:10.562526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.562443    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqb5\" (UniqueName: \"kubernetes.io/projected/3200469f-32a5-4998-b373-23fc735afe4b-kube-api-access-lbqb5\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.562742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.562560    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.562742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.562619    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.563271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.563245    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.564964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.564942    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.625298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.625263    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqb5\" (UniqueName: \"kubernetes.io/projected/3200469f-32a5-4998-b373-23fc735afe4b-kube-api-access-lbqb5\") pod \"maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:10.732559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:10.732517    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:11.109732 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:25:11.109697    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3200469f_32a5_4998_b373_23fc735afe4b.slice/crio-c439f536a56f7b75aaa64dcd702e801ef5918bffff4e987a4def2399de8fbbc8 WatchSource:0}: Error finding container c439f536a56f7b75aaa64dcd702e801ef5918bffff4e987a4def2399de8fbbc8: Status 404 returned error can't find the container with id c439f536a56f7b75aaa64dcd702e801ef5918bffff4e987a4def2399de8fbbc8
Oct 30 22:25:11.112675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.112578    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"]
Oct 30 22:25:11.303984 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.303947    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m" event={"ID":"3200469f-32a5-4998-b373-23fc735afe4b","Type":"ContainerStarted","Data":"c439f536a56f7b75aaa64dcd702e801ef5918bffff4e987a4def2399de8fbbc8"}
Oct 30 22:25:11.442969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.442947    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:11.573823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.573794    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-openshift-ca\") pod \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") "
Oct 30 22:25:11.574013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.573892    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-xks-ca\") pod \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") "
Oct 30 22:25:11.574013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.573974    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-kube-api-access-w9pf6\" (UniqueName: \"kubernetes.io/projected/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-kube-api-access-w9pf6\") pod \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\" (UID: \"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df\") "
Oct 30 22:25:11.574210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.574185    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-openshift-ca" pod "dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" (UID: "dd9f5047-0b99-4ddc-9482-3cfdcb2b26df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:11.574375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.574358    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:11.576049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.576024    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-xks-ca" pod "dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" (UID: "dd9f5047-0b99-4ddc-9482-3cfdcb2b26df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:11.576193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.576178    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-kube-api-access-w9pf6" pod "dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" (UID: "dd9f5047-0b99-4ddc-9482-3cfdcb2b26df"). InnerVolumeSpecName "kube-api-access-w9pf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:11.676005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.675973    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:11.676005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:11.676004    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9pf6\" (UniqueName: \"kubernetes.io/projected/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df-kube-api-access-w9pf6\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:12.310679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:12.310629    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn" event={"ID":"dd9f5047-0b99-4ddc-9482-3cfdcb2b26df","Type":"ContainerDied","Data":"27e65ec19ef7681cb8fa7176a99f7fed1d528df939626c840f4a0ec6041865d7"}
Oct 30 22:25:12.310679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:12.310663    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"
Oct 30 22:25:12.311168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:12.310702    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27e65ec19ef7681cb8fa7176a99f7fed1d528df939626c840f4a0ec6041865d7"
Oct 30 22:25:12.312310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:12.312282    5503 generic.go:356] "Generic (PLEG): container finished" podID="3200469f-32a5-4998-b373-23fc735afe4b" containerID="2e6e8a4106a9fea4b1e820a6cdfefb6244bc07349329f4821769066968357999" exitCode=0
Oct 30 22:25:12.312444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:12.312370    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m" event={"ID":"3200469f-32a5-4998-b373-23fc735afe4b","Type":"ContainerDied","Data":"2e6e8a4106a9fea4b1e820a6cdfefb6244bc07349329f4821769066968357999"}
Oct 30 22:25:13.454110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.454086    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:13.599169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.599090    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-xks-ca\") pod \"3200469f-32a5-4998-b373-23fc735afe4b\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") "
Oct 30 22:25:13.599342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.599255    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3200469f-32a5-4998-b373-23fc735afe4b-kube-api-access-lbqb5\" (UniqueName: \"kubernetes.io/projected/3200469f-32a5-4998-b373-23fc735afe4b-kube-api-access-lbqb5\") pod \"3200469f-32a5-4998-b373-23fc735afe4b\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") "
Oct 30 22:25:13.599466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.599439    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-openshift-ca\") pod \"3200469f-32a5-4998-b373-23fc735afe4b\" (UID: \"3200469f-32a5-4998-b373-23fc735afe4b\") "
Oct 30 22:25:13.599948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.599866    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-openshift-ca" pod "3200469f-32a5-4998-b373-23fc735afe4b" (UID: "3200469f-32a5-4998-b373-23fc735afe4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:13.601267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.601237    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-xks-ca" pod "3200469f-32a5-4998-b373-23fc735afe4b" (UID: "3200469f-32a5-4998-b373-23fc735afe4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:13.601407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.601388    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3200469f-32a5-4998-b373-23fc735afe4b-kube-api-access-lbqb5" pod "3200469f-32a5-4998-b373-23fc735afe4b" (UID: "3200469f-32a5-4998-b373-23fc735afe4b"). InnerVolumeSpecName "kube-api-access-lbqb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:13.701607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.701571    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:13.701607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.701602    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3200469f-32a5-4998-b373-23fc735afe4b-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:13.701835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:13.701612    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqb5\" (UniqueName: \"kubernetes.io/projected/3200469f-32a5-4998-b373-23fc735afe4b-kube-api-access-lbqb5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:14.284877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:14.284847    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:14.358092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:14.358051    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m" event={"ID":"3200469f-32a5-4998-b373-23fc735afe4b","Type":"ContainerDied","Data":"c439f536a56f7b75aaa64dcd702e801ef5918bffff4e987a4def2399de8fbbc8"}
Oct 30 22:25:14.358660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:14.358382    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c439f536a56f7b75aaa64dcd702e801ef5918bffff4e987a4def2399de8fbbc8"
Oct 30 22:25:14.358660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:14.358478    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"
Oct 30 22:25:14.638931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:14.638844    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:25:19.202603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:19.202558    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:20.055934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:20.055894    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.260481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.260445    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.478320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.478277    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.738469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.738435    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:22.949135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.949106    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"]
Oct 30 22:25:22.949593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.949580    5503 cpu_manager.go:414] "containerMap: removing container" podUID="dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" containerName="revoke-keys"
Oct 30 22:25:22.949674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.949595    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" containerName="revoke-keys"
Oct 30 22:25:22.949674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.949619    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3200469f-32a5-4998-b373-23fc735afe4b" containerName="revoke-keys"
Oct 30 22:25:22.949674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.949625    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3200469f-32a5-4998-b373-23fc735afe4b" containerName="revoke-keys"
Oct 30 22:25:22.949824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.949740    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3200469f-32a5-4998-b373-23fc735afe4b" containerName="revoke-keys"
Oct 30 22:25:22.949824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.949757    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" containerName="revoke-keys"
Oct 30 22:25:22.953581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:22.953565    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.008263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.008187    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:23.009724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.009696    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"]
Oct 30 22:25:23.056958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.056929    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:25:23.057113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.056991    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:25:23.059051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.059030    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:25:23.142980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.142951    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzlj\" (UniqueName: \"kubernetes.io/projected/d481116c-fbad-45fd-b057-96215bebb3b9-kube-api-access-6qzlj\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.143156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.142994    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.143156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.143019    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.244216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.244172    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzlj\" (UniqueName: \"kubernetes.io/projected/d481116c-fbad-45fd-b057-96215bebb3b9-kube-api-access-6qzlj\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.244420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.244232    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.244420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.244263    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.244992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.244963    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.246615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.246591    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.313983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.313891    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzlj\" (UniqueName: \"kubernetes.io/projected/d481116c-fbad-45fd-b057-96215bebb3b9-kube-api-access-6qzlj\") pod \"maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.566476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.566397    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:23.924446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:23.924413    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:23.999098 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:25:23.999068    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd481116c_fbad_45fd_b057_96215bebb3b9.slice/crio-2b367b3933008d97c8974ace3dc9dd0f6d003ad06f09073e6e7e6b4e0e2c204d WatchSource:0}: Error finding container 2b367b3933008d97c8974ace3dc9dd0f6d003ad06f09073e6e7e6b4e0e2c204d: Status 404 returned error can't find the container with id 2b367b3933008d97c8974ace3dc9dd0f6d003ad06f09073e6e7e6b4e0e2c204d
Oct 30 22:25:24.002540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:24.002516    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"]
Oct 30 22:25:24.372092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:24.372057    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:24.427046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:24.427013    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs" event={"ID":"d481116c-fbad-45fd-b057-96215bebb3b9","Type":"ContainerStarted","Data":"f4f99cd6667a985b5f66f100fc0b2d5a8881499a2f81f39467cc10d5b1df10f3"}
Oct 30 22:25:24.427209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:24.427069    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs" event={"ID":"d481116c-fbad-45fd-b057-96215bebb3b9","Type":"ContainerStarted","Data":"2b367b3933008d97c8974ace3dc9dd0f6d003ad06f09073e6e7e6b4e0e2c204d"}
Oct 30 22:25:24.548354 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:24.548256    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs" podStartSLOduration=2.548243125 podStartE2EDuration="2.548243125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:24.545674539 +0000 UTC m=+2300.880251318" watchObservedRunningTime="2026-10-30 22:25:24.548243125 +0000 UTC m=+2300.882819895"
Oct 30 22:25:25.435122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:25.435088    5503 generic.go:356] "Generic (PLEG): container finished" podID="d481116c-fbad-45fd-b057-96215bebb3b9" containerID="f4f99cd6667a985b5f66f100fc0b2d5a8881499a2f81f39467cc10d5b1df10f3" exitCode=0
Oct 30 22:25:25.435493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:25.435180    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs" event={"ID":"d481116c-fbad-45fd-b057-96215bebb3b9","Type":"ContainerDied","Data":"f4f99cd6667a985b5f66f100fc0b2d5a8881499a2f81f39467cc10d5b1df10f3"}
Oct 30 22:25:25.672218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:25.672180    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:26.587903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.587879    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:26.692179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.692103    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-openshift-ca\") pod \"d481116c-fbad-45fd-b057-96215bebb3b9\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") "
Oct 30 22:25:26.692179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.692153    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d481116c-fbad-45fd-b057-96215bebb3b9-kube-api-access-6qzlj\" (UniqueName: \"kubernetes.io/projected/d481116c-fbad-45fd-b057-96215bebb3b9-kube-api-access-6qzlj\") pod \"d481116c-fbad-45fd-b057-96215bebb3b9\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") "
Oct 30 22:25:26.692369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.692331    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-xks-ca\") pod \"d481116c-fbad-45fd-b057-96215bebb3b9\" (UID: \"d481116c-fbad-45fd-b057-96215bebb3b9\") "
Oct 30 22:25:26.692441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.692419    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-openshift-ca" pod "d481116c-fbad-45fd-b057-96215bebb3b9" (UID: "d481116c-fbad-45fd-b057-96215bebb3b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:26.692780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.692761    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:26.694389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.694363    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-xks-ca" pod "d481116c-fbad-45fd-b057-96215bebb3b9" (UID: "d481116c-fbad-45fd-b057-96215bebb3b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:26.694389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.694382    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d481116c-fbad-45fd-b057-96215bebb3b9-kube-api-access-6qzlj" pod "d481116c-fbad-45fd-b057-96215bebb3b9" (UID: "d481116c-fbad-45fd-b057-96215bebb3b9"). InnerVolumeSpecName "kube-api-access-6qzlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:26.793886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.793857    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481116c-fbad-45fd-b057-96215bebb3b9-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:26.793886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:26.793883    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzlj\" (UniqueName: \"kubernetes.io/projected/d481116c-fbad-45fd-b057-96215bebb3b9-kube-api-access-6qzlj\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:27.449598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:27.449566    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"
Oct 30 22:25:27.449903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:27.449600    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs" event={"ID":"d481116c-fbad-45fd-b057-96215bebb3b9","Type":"ContainerDied","Data":"2b367b3933008d97c8974ace3dc9dd0f6d003ad06f09073e6e7e6b4e0e2c204d"}
Oct 30 22:25:27.449903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:27.449679    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b367b3933008d97c8974ace3dc9dd0f6d003ad06f09073e6e7e6b4e0e2c204d"
Oct 30 22:25:28.099193 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:25:28.099150    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:25:30.911176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:30.911133    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:34.890459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:34.890417    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:35.084836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:35.079667    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:37.470398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:37.470364    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:42.460165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:42.460129    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:46.941659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:46.928508    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:48.391692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:48.391656    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:48.595300 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:48.595247    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:48.880359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:48.880326    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:49.026644 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.026601    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"]
Oct 30 22:25:49.077987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.077951    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"]
Oct 30 22:25:49.078246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.078216    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" containerID="cri-o://cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118" gracePeriod=30
Oct 30 22:25:49.080682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.080656    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3bd7fe-xxm6m"]
Oct 30 22:25:49.368127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.368092    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:25:49.497087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.497050    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:49.789563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.789529    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3200469f-32a5-4998-b373-23fc735afe4b" path="/var/lib/kubelet/pods/3200469f-32a5-4998-b373-23fc735afe4b/volumes"
Oct 30 22:25:49.902525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:49.902493    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:50.322553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:50.322517    5503 patch_prober.go:29] interesting pod/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:25:50.322915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:50.322879    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:25:50.551752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:50.551713    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:50.773421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:50.773380    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:53.373789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:53.373759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.341592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.341537    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:25:54.438584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438550    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-kube-api-access-g4xw8\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-kube-api-access-g4xw8\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438602    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-credential-socket\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438622    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-socket\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438672    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-podinfo\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438698    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istiod-ca-cert\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438715    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-data\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438788    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-envoy\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438821    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-certs\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438890    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-token\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-token\") pod \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\" (UID: \"138d9a0c-5eb8-4daf-9b29-ed780c71d938\") "
Oct 30 22:25:54.439056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.438905    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-credential-socket" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.439522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439070    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-data" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.439522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439161    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-certs" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.439522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439246    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-socket" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.439522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439288    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istiod-ca-cert" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:54.439736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439558    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.439736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439581    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.439736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439596    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.439736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439611    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.439736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.439625    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.440988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.440964    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-kube-api-access-g4xw8" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "kube-api-access-g4xw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:54.441107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.441083    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-token" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:54.441394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.441371    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-podinfo" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:54.441504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.441391    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-envoy" pod "138d9a0c-5eb8-4daf-9b29-ed780c71d938" (UID: "138d9a0c-5eb8-4daf-9b29-ed780c71d938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:54.541117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.541083    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.541117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.541111    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.541117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.541121    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4xw8\" (UniqueName: \"kubernetes.io/projected/138d9a0c-5eb8-4daf-9b29-ed780c71d938-kube-api-access-g4xw8\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.541364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.541132    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d9a0c-5eb8-4daf-9b29-ed780c71d938-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:54.713373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.713337    5503 generic.go:356] "Generic (PLEG): container finished" podID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerID="cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118" exitCode=0
Oct 30 22:25:54.713607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.713385    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"
Oct 30 22:25:54.713607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.713422    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" event={"ID":"138d9a0c-5eb8-4daf-9b29-ed780c71d938","Type":"ContainerDied","Data":"cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118"}
Oct 30 22:25:54.713607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.713477    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr" event={"ID":"138d9a0c-5eb8-4daf-9b29-ed780c71d938","Type":"ContainerDied","Data":"ce59566e5182fd50f938e248cb3d89067e45ff9b0b7ad844666f2dfe107e460c"}
Oct 30 22:25:54.713607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.713502    5503 scope.go:122] "RemoveContainer" containerID="cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118"
Oct 30 22:25:54.724861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.724842    5503 scope.go:122] "RemoveContainer" containerID="cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118"
Oct 30 22:25:54.725116 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:25:54.725098    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118\": container with ID starting with cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118 not found: ID does not exist" containerID="cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118"
Oct 30 22:25:54.725159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.725125    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118"} err="failed to get container status \"cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118\": rpc error: code = NotFound desc = could not find container \"cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118\": container with ID starting with cb6bafb590231ef882739600e3385cbcb411d976d1f9436cee8a295495e85118 not found: ID does not exist"
Oct 30 22:25:54.741751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.741728    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:54.899084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.899044    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"]
Oct 30 22:25:54.914821 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:54.914788    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7f66ab-openshift-default-5cbbd48869-nplkr"]
Oct 30 22:25:55.030065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:55.029986    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:55.789401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:55.789366    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" path="/var/lib/kubelet/pods/138d9a0c-5eb8-4daf-9b29-ed780c71d938/volumes"
Oct 30 22:25:56.183145 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:56.183114    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.374147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:56.374117    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:56.556818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:56.556736    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:57.958774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:57.958740    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:58.125360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:58.125312    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:25:58.142487 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:25:58.142454    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:25:58.531996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:25:58.531960    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:01.157027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:01.156992    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:02.342326 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.342290    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"]
Oct 30 22:26:02.342806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.342791    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d481116c-fbad-45fd-b057-96215bebb3b9" containerName="revoke-keys"
Oct 30 22:26:02.342852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.342807    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d481116c-fbad-45fd-b057-96215bebb3b9" containerName="revoke-keys"
Oct 30 22:26:02.342852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.342828    5503 cpu_manager.go:414] "containerMap: removing container" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy"
Oct 30 22:26:02.342852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.342833    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy"
Oct 30 22:26:02.342950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.342919    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="138d9a0c-5eb8-4daf-9b29-ed780c71d938" containerName="istio-proxy"
Oct 30 22:26:02.342950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.342936    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d481116c-fbad-45fd-b057-96215bebb3b9" containerName="revoke-keys"
Oct 30 22:26:02.346194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.346179    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.376476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.376442    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2xqc\" (UniqueName: \"kubernetes.io/projected/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-kube-api-access-l2xqc\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.376626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.376604    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.376761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.376744    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.400567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.400532    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"]
Oct 30 22:26:02.449018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.448966    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:02.449210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.449084    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:02.449210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.449087    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:26:02.449210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.449162    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:02.477373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.477341    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.477525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.477411    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2xqc\" (UniqueName: \"kubernetes.io/projected/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-kube-api-access-l2xqc\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.477579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.477535    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.478297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.478269    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.479893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.479862    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.540927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.540889    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2xqc\" (UniqueName: \"kubernetes.io/projected/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-kube-api-access-l2xqc\") pod \"maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.658907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.658876    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:02.693679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.693618    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"]
Oct 30 22:26:02.697650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.697616    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.755166 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.755134    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"]
Oct 30 22:26:02.782176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.782146    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.782176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.782178    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2m9\" (UniqueName: \"kubernetes.io/projected/3d142ab9-3676-4a91-a470-de5c79226d33-kube-api-access-gn2m9\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.782401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.782309    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.883466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.883431    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.883694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.883668    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.883781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.883697    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2m9\" (UniqueName: \"kubernetes.io/projected/3d142ab9-3676-4a91-a470-de5c79226d33-kube-api-access-gn2m9\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.884254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.884223    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.885875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.885853    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:02.960109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:02.960038    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2m9\" (UniqueName: \"kubernetes.io/projected/3d142ab9-3676-4a91-a470-de5c79226d33-kube-api-access-gn2m9\") pod \"maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:03.012201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.012168    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:03.032006 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:26:03.031974    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-a32da6249803344d9998b41a37601e19bde070f947ee9aa3f180f5946e6bff78 WatchSource:0}: Error finding container a32da6249803344d9998b41a37601e19bde070f947ee9aa3f180f5946e6bff78: Status 404 returned error can't find the container with id a32da6249803344d9998b41a37601e19bde070f947ee9aa3f180f5946e6bff78
Oct 30 22:26:03.034083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.034062    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"]
Oct 30 22:26:03.385767 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:26:03.385728    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d142ab9_3676_4a91_a470_de5c79226d33.slice/crio-3ffb83b8f81ecd65def54fe9a77de2a9286c9bc4e327c2e23cebd72e5a1777f7 WatchSource:0}: Error finding container 3ffb83b8f81ecd65def54fe9a77de2a9286c9bc4e327c2e23cebd72e5a1777f7: Status 404 returned error can't find the container with id 3ffb83b8f81ecd65def54fe9a77de2a9286c9bc4e327c2e23cebd72e5a1777f7
Oct 30 22:26:03.388082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.388057    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"]
Oct 30 22:26:03.407076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.407055    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:03.829236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.829140    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd" event={"ID":"b5a0393b-7a31-46a5-ab6f-9143f2d72a11","Type":"ContainerStarted","Data":"c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119"}
Oct 30 22:26:03.829236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.829203    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd" event={"ID":"b5a0393b-7a31-46a5-ab6f-9143f2d72a11","Type":"ContainerStarted","Data":"a32da6249803344d9998b41a37601e19bde070f947ee9aa3f180f5946e6bff78"}
Oct 30 22:26:03.830837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.830808    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr" event={"ID":"3d142ab9-3676-4a91-a470-de5c79226d33","Type":"ContainerStarted","Data":"62a2b5e212cac7d5465261c99fadec6df832e9cda8776a6406bde2c01f666391"}
Oct 30 22:26:03.830968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.830852    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr" event={"ID":"3d142ab9-3676-4a91-a470-de5c79226d33","Type":"ContainerStarted","Data":"3ffb83b8f81ecd65def54fe9a77de2a9286c9bc4e327c2e23cebd72e5a1777f7"}
Oct 30 22:26:03.947454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:03.947401    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd" podStartSLOduration=1.947389598 podStartE2EDuration="1.947389598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:03.946690051 +0000 UTC m=+2340.281266821" watchObservedRunningTime="2026-10-30 22:26:03.947389598 +0000 UTC m=+2340.281966366"
Oct 30 22:26:04.066214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.066156    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr" podStartSLOduration=2.066138446 podStartE2EDuration="2.066138446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:04.064800144 +0000 UTC m=+2340.399376913" watchObservedRunningTime="2026-10-30 22:26:04.066138446 +0000 UTC m=+2340.400715215"
Oct 30 22:26:04.494838 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:04.494755    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:04.838431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.838346    5503 generic.go:356] "Generic (PLEG): container finished" podID="3d142ab9-3676-4a91-a470-de5c79226d33" containerID="62a2b5e212cac7d5465261c99fadec6df832e9cda8776a6406bde2c01f666391" exitCode=0
Oct 30 22:26:04.838616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.838435    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr" event={"ID":"3d142ab9-3676-4a91-a470-de5c79226d33","Type":"ContainerDied","Data":"62a2b5e212cac7d5465261c99fadec6df832e9cda8776a6406bde2c01f666391"}
Oct 30 22:26:04.840315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.840292    5503 generic.go:356] "Generic (PLEG): container finished" podID="b5a0393b-7a31-46a5-ab6f-9143f2d72a11" containerID="c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119" exitCode=0
Oct 30 22:26:04.840434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.840354    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd" event={"ID":"b5a0393b-7a31-46a5-ab6f-9143f2d72a11","Type":"ContainerDied","Data":"c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119"}
Oct 30 22:26:04.868984 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.868962    5503 scope.go:122] "RemoveContainer" containerID="8dac9e8169e8fc1b7bdb88988393dc58a4192a85c491110d1ebe710be22fbc5a"
Oct 30 22:26:04.881008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.880984    5503 scope.go:122] "RemoveContainer" containerID="2166d98296981decdfac4666d034753d960ad6981170eb8f30e67b41fdfa0be4"
Oct 30 22:26:04.889707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.889679    5503 scope.go:122] "RemoveContainer" containerID="08c8d700e57fd9bafc03d12a07c011b8de075c922c7b0e235dfca76be8501c60"
Oct 30 22:26:04.898001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:04.897984    5503 scope.go:122] "RemoveContainer" containerID="0d86cdf5a9d68ef4f6b2edaf2e6786e675d3df483a6882b1ac9842ddac3f0290"
Oct 30 22:26:06.036781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.036751    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:06.048329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.048308    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:06.117151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117086    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-kube-api-access-l2xqc\" (UniqueName: \"kubernetes.io/projected/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-kube-api-access-l2xqc\") pod \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") "
Oct 30 22:26:06.117151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117129    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-xks-ca\") pod \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") "
Oct 30 22:26:06.117336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117166    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-openshift-ca\") pod \"3d142ab9-3676-4a91-a470-de5c79226d33\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") "
Oct 30 22:26:06.117336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117266    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-xks-ca\") pod \"3d142ab9-3676-4a91-a470-de5c79226d33\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") "
Oct 30 22:26:06.117454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117402    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d142ab9-3676-4a91-a470-de5c79226d33-kube-api-access-gn2m9\" (UniqueName: \"kubernetes.io/projected/3d142ab9-3676-4a91-a470-de5c79226d33-kube-api-access-gn2m9\") pod \"3d142ab9-3676-4a91-a470-de5c79226d33\" (UID: \"3d142ab9-3676-4a91-a470-de5c79226d33\") "
Oct 30 22:26:06.117454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117420    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-openshift-ca" pod "3d142ab9-3676-4a91-a470-de5c79226d33" (UID: "3d142ab9-3676-4a91-a470-de5c79226d33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:06.117576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117546    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-openshift-ca\") pod \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\" (UID: \"b5a0393b-7a31-46a5-ab6f-9143f2d72a11\") "
Oct 30 22:26:06.117936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.117909    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-openshift-ca" pod "b5a0393b-7a31-46a5-ab6f-9143f2d72a11" (UID: "b5a0393b-7a31-46a5-ab6f-9143f2d72a11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:06.118080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.118065    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.118151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.118082    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.119375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.119352    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-xks-ca" pod "3d142ab9-3676-4a91-a470-de5c79226d33" (UID: "3d142ab9-3676-4a91-a470-de5c79226d33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:06.119458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.119440    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d142ab9-3676-4a91-a470-de5c79226d33-kube-api-access-gn2m9" pod "3d142ab9-3676-4a91-a470-de5c79226d33" (UID: "3d142ab9-3676-4a91-a470-de5c79226d33"). InnerVolumeSpecName "kube-api-access-gn2m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:06.119657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.119622    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-xks-ca" pod "b5a0393b-7a31-46a5-ab6f-9143f2d72a11" (UID: "b5a0393b-7a31-46a5-ab6f-9143f2d72a11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:06.119717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.119666    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-kube-api-access-l2xqc" pod "b5a0393b-7a31-46a5-ab6f-9143f2d72a11" (UID: "b5a0393b-7a31-46a5-ab6f-9143f2d72a11"). InnerVolumeSpecName "kube-api-access-l2xqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:06.219225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.219193    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2m9\" (UniqueName: \"kubernetes.io/projected/3d142ab9-3676-4a91-a470-de5c79226d33-kube-api-access-gn2m9\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.219225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.219220    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2xqc\" (UniqueName: \"kubernetes.io/projected/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-kube-api-access-l2xqc\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.219225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.219230    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5a0393b-7a31-46a5-ab6f-9143f2d72a11-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.219441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.219244    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d142ab9-3676-4a91-a470-de5c79226d33-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:06.852581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.852547    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"
Oct 30 22:26:06.852815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.852576    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd" event={"ID":"b5a0393b-7a31-46a5-ab6f-9143f2d72a11","Type":"ContainerDied","Data":"a32da6249803344d9998b41a37601e19bde070f947ee9aa3f180f5946e6bff78"}
Oct 30 22:26:06.852815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.852659    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32da6249803344d9998b41a37601e19bde070f947ee9aa3f180f5946e6bff78"
Oct 30 22:26:06.854061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.854039    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr" event={"ID":"3d142ab9-3676-4a91-a470-de5c79226d33","Type":"ContainerDied","Data":"3ffb83b8f81ecd65def54fe9a77de2a9286c9bc4e327c2e23cebd72e5a1777f7"}
Oct 30 22:26:06.854061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.854053    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"
Oct 30 22:26:06.854193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:06.854073    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ffb83b8f81ecd65def54fe9a77de2a9286c9bc4e327c2e23cebd72e5a1777f7"
Oct 30 22:26:08.195561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:08.195528    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:08.349899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:08.349860    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:10.964003 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:10.963967    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:12.289566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:12.289496    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.379158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:13.379071    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.725120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:13.725086    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:13.895831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:13.895800    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.049138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:14.049043    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.187147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:14.187110    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"]
Oct 30 22:26:14.241705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:14.220334    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-40fb42-8z2hs"]
Oct 30 22:26:14.330548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:14.330457    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.487379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:14.487334    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.648861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:14.648773    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:14.954297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:14.954268    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.273260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:15.273166    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.427214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:15.427181    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:15.463074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:15.463041    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"]
Oct 30 22:26:15.513750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:15.513728    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0e90fe-9pw28"]
Oct 30 22:26:15.813548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:15.803778    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51848ac5-bd14-4858-bbe9-756075fac717" path="/var/lib/kubelet/pods/51848ac5-bd14-4858-bbe9-756075fac717/volumes"
Oct 30 22:26:15.813548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:15.806039    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d481116c-fbad-45fd-b057-96215bebb3b9" path="/var/lib/kubelet/pods/d481116c-fbad-45fd-b057-96215bebb3b9/volumes"
Oct 30 22:26:15.909177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:15.909141    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:18.684707 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:18.684621    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:18.685106 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:18.684779    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:19.487250 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:19.487206    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:20.041390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:20.041356    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:20.852324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:20.852291    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"]
Oct 30 22:26:20.983027 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:20.982981    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:21.028481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:21.028435    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:21.355083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:21.355048    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:22.278102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.278069    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:22.615270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.615190    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"]
Oct 30 22:26:22.615723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.615708    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3d142ab9-3676-4a91-a470-de5c79226d33" containerName="revoke-keys"
Oct 30 22:26:22.615778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.615723    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d142ab9-3676-4a91-a470-de5c79226d33" containerName="revoke-keys"
Oct 30 22:26:22.615778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.615741    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b5a0393b-7a31-46a5-ab6f-9143f2d72a11" containerName="revoke-keys"
Oct 30 22:26:22.615778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.615747    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a0393b-7a31-46a5-ab6f-9143f2d72a11" containerName="revoke-keys"
Oct 30 22:26:22.615879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.615818    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d142ab9-3676-4a91-a470-de5c79226d33" containerName="revoke-keys"
Oct 30 22:26:22.615879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.615828    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a0393b-7a31-46a5-ab6f-9143f2d72a11" containerName="revoke-keys"
Oct 30 22:26:22.620241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.620217    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.655829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.655801    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.655998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.655849    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.655998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.655947    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnzb\" (UniqueName: \"kubernetes.io/projected/1fa96caa-74e1-45e1-a698-3fb9f5631a54-kube-api-access-fnnzb\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.674716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.674688    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:22.677828 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.677799    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"]
Oct 30 22:26:22.725394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.725362    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:22.752183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.752146    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:22.752351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.752206    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:26:22.756780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.756755    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.756842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.756802    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.756940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.756924    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnzb\" (UniqueName: \"kubernetes.io/projected/1fa96caa-74e1-45e1-a698-3fb9f5631a54-kube-api-access-fnnzb\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.757403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.757383    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.759169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.759150    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.819846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.819814    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnzb\" (UniqueName: \"kubernetes.io/projected/1fa96caa-74e1-45e1-a698-3fb9f5631a54-kube-api-access-fnnzb\") pod \"maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:22.933832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:22.933799    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:23.057944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:23.057889    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:23.304764 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:26:23.304732    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa96caa_74e1_45e1_a698_3fb9f5631a54.slice/crio-cfcf6f180ff2ab4b21c0d0f331ee7196370c953a0b783ce96539ce38d9c9ebef WatchSource:0}: Error finding container cfcf6f180ff2ab4b21c0d0f331ee7196370c953a0b783ce96539ce38d9c9ebef: Status 404 returned error can't find the container with id cfcf6f180ff2ab4b21c0d0f331ee7196370c953a0b783ce96539ce38d9c9ebef
Oct 30 22:26:23.307282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:23.307259    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"]
Oct 30 22:26:23.979646 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:23.979609    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp" event={"ID":"1fa96caa-74e1-45e1-a698-3fb9f5631a54","Type":"ContainerStarted","Data":"89934caa9e0281c4df06ab28c1985b358000877029b1ffd040dd68c3a692f663"}
Oct 30 22:26:23.980035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:23.979687    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp" event={"ID":"1fa96caa-74e1-45e1-a698-3fb9f5631a54","Type":"ContainerStarted","Data":"cfcf6f180ff2ab4b21c0d0f331ee7196370c953a0b783ce96539ce38d9c9ebef"}
Oct 30 22:26:24.099162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:24.099112    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp" podStartSLOduration=2.099097464 podStartE2EDuration="2.099097464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:24.096932506 +0000 UTC m=+2360.431509275" watchObservedRunningTime="2026-10-30 22:26:24.099097464 +0000 UTC m=+2360.433674232"
Oct 30 22:26:24.222121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:24.222063    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:24.911043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:24.911013    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:24.986866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:24.986831    5503 generic.go:356] "Generic (PLEG): container finished" podID="1fa96caa-74e1-45e1-a698-3fb9f5631a54" containerID="89934caa9e0281c4df06ab28c1985b358000877029b1ffd040dd68c3a692f663" exitCode=0
Oct 30 22:26:24.987320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:24.986884    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp" event={"ID":"1fa96caa-74e1-45e1-a698-3fb9f5631a54","Type":"ContainerDied","Data":"89934caa9e0281c4df06ab28c1985b358000877029b1ffd040dd68c3a692f663"}
Oct 30 22:26:25.068889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:25.068851    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:25.230778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:25.230689    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:26.123441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.123417    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:26.197687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.197627    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-xks-ca\") pod \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") "
Oct 30 22:26:26.197865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.197715    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa96caa-74e1-45e1-a698-3fb9f5631a54-kube-api-access-fnnzb\" (UniqueName: \"kubernetes.io/projected/1fa96caa-74e1-45e1-a698-3fb9f5631a54-kube-api-access-fnnzb\") pod \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") "
Oct 30 22:26:26.197865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.197762    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-openshift-ca\") pod \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\" (UID: \"1fa96caa-74e1-45e1-a698-3fb9f5631a54\") "
Oct 30 22:26:26.198159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.198135    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-openshift-ca" pod "1fa96caa-74e1-45e1-a698-3fb9f5631a54" (UID: "1fa96caa-74e1-45e1-a698-3fb9f5631a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:26.199770 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.199750    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-xks-ca" pod "1fa96caa-74e1-45e1-a698-3fb9f5631a54" (UID: "1fa96caa-74e1-45e1-a698-3fb9f5631a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:26.199900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.199880    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa96caa-74e1-45e1-a698-3fb9f5631a54-kube-api-access-fnnzb" pod "1fa96caa-74e1-45e1-a698-3fb9f5631a54" (UID: "1fa96caa-74e1-45e1-a698-3fb9f5631a54"). InnerVolumeSpecName "kube-api-access-fnnzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:26.299848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.299774    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnzb\" (UniqueName: \"kubernetes.io/projected/1fa96caa-74e1-45e1-a698-3fb9f5631a54-kube-api-access-fnnzb\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:26.299848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.299802    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:26.299848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.299813    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa96caa-74e1-45e1-a698-3fb9f5631a54-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:26.999394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.999365    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"
Oct 30 22:26:26.999582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.999398    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp" event={"ID":"1fa96caa-74e1-45e1-a698-3fb9f5631a54","Type":"ContainerDied","Data":"cfcf6f180ff2ab4b21c0d0f331ee7196370c953a0b783ce96539ce38d9c9ebef"}
Oct 30 22:26:26.999582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:26.999445    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfcf6f180ff2ab4b21c0d0f331ee7196370c953a0b783ce96539ce38d9c9ebef"
Oct 30 22:26:28.098958 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:28.098918    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:26:31.048341 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:31.048307    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:31.634720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:31.634666    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:34.335715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:34.335679    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"]
Oct 30 22:26:34.474199 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:34.474160    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:34.677593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:34.677552    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:34.830999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:34.830963    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:35.661841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:35.661808    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:35.872584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:35.872516    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm" podUID="1024f0de-a385-4afd-aed8-91259a20574f" containerName="main" containerID="cri-o://beb198bf2bc1c57bcf3a2f5f2e5ec924ae182923cd6616f92f90595e04920869" gracePeriod=45
Oct 30 22:26:36.060832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.060804    5503 generic.go:356] "Generic (PLEG): container finished" podID="1024f0de-a385-4afd-aed8-91259a20574f" containerID="beb198bf2bc1c57bcf3a2f5f2e5ec924ae182923cd6616f92f90595e04920869" exitCode=0
Oct 30 22:26:36.060982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.060875    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm" event={"ID":"1024f0de-a385-4afd-aed8-91259a20574f","Type":"ContainerDied","Data":"beb198bf2bc1c57bcf3a2f5f2e5ec924ae182923cd6616f92f90595e04920869"}
Oct 30 22:26:36.146391 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.146355    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.159722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.159699    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:26:36.302746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.302669    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:36.348844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.348816    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-dshm\") pod \"1024f0de-a385-4afd-aed8-91259a20574f\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") "
Oct 30 22:26:36.349025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.348874    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1024f0de-a385-4afd-aed8-91259a20574f-kube-api-access-vxd7m\" (UniqueName: \"kubernetes.io/projected/1024f0de-a385-4afd-aed8-91259a20574f-kube-api-access-vxd7m\") pod \"1024f0de-a385-4afd-aed8-91259a20574f\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") "
Oct 30 22:26:36.349025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.348927    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-home\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-home\") pod \"1024f0de-a385-4afd-aed8-91259a20574f\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") "
Oct 30 22:26:36.349025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.348955    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-model-cache\") pod \"1024f0de-a385-4afd-aed8-91259a20574f\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") "
Oct 30 22:26:36.349025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.348972    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-tmp-dir\") pod \"1024f0de-a385-4afd-aed8-91259a20574f\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") "
Oct 30 22:26:36.349266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.349107    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1024f0de-a385-4afd-aed8-91259a20574f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1024f0de-a385-4afd-aed8-91259a20574f-tls-certs\") pod \"1024f0de-a385-4afd-aed8-91259a20574f\" (UID: \"1024f0de-a385-4afd-aed8-91259a20574f\") "
Oct 30 22:26:36.349266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.349185    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-model-cache" pod "1024f0de-a385-4afd-aed8-91259a20574f" (UID: "1024f0de-a385-4afd-aed8-91259a20574f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:36.349266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.349197    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-home" pod "1024f0de-a385-4afd-aed8-91259a20574f" (UID: "1024f0de-a385-4afd-aed8-91259a20574f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:36.349444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.349270    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-tmp-dir" pod "1024f0de-a385-4afd-aed8-91259a20574f" (UID: "1024f0de-a385-4afd-aed8-91259a20574f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:36.349596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.349575    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.349596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.349598    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.349596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.349608    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.351195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.351171    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1024f0de-a385-4afd-aed8-91259a20574f-kube-api-access-vxd7m" pod "1024f0de-a385-4afd-aed8-91259a20574f" (UID: "1024f0de-a385-4afd-aed8-91259a20574f"). InnerVolumeSpecName "kube-api-access-vxd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:36.351527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.351505    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1024f0de-a385-4afd-aed8-91259a20574f-tls-certs" pod "1024f0de-a385-4afd-aed8-91259a20574f" (UID: "1024f0de-a385-4afd-aed8-91259a20574f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:36.351527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.351518    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-dshm" pod "1024f0de-a385-4afd-aed8-91259a20574f" (UID: "1024f0de-a385-4afd-aed8-91259a20574f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:36.450574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.450532    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1024f0de-a385-4afd-aed8-91259a20574f-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.450574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.450569    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1024f0de-a385-4afd-aed8-91259a20574f-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.450574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:36.450579    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxd7m\" (UniqueName: \"kubernetes.io/projected/1024f0de-a385-4afd-aed8-91259a20574f-kube-api-access-vxd7m\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:37.068698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.068666    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm" event={"ID":"1024f0de-a385-4afd-aed8-91259a20574f","Type":"ContainerDied","Data":"f90880d82df5d7878158a9eeec20e060781a930eec75c633da66ae7083026fd9"}
Oct 30 22:26:37.068698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.068698    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"
Oct 30 22:26:37.069236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.068715    5503 scope.go:122] "RemoveContainer" containerID="beb198bf2bc1c57bcf3a2f5f2e5ec924ae182923cd6616f92f90595e04920869"
Oct 30 22:26:37.154556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.154521    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.238691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.238655    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"]
Oct 30 22:26:37.241111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.241087    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0e90fe/ipp-route-0e90fe-kserve-64c675dbfc-tmhdm"]
Oct 30 22:26:37.419967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.419937    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.572488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.572457    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:37.610578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.610540    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"]
Oct 30 22:26:37.612829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.612806    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-872f4a-dq6qr"]
Oct 30 22:26:37.789618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.789539    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1024f0de-a385-4afd-aed8-91259a20574f" path="/var/lib/kubelet/pods/1024f0de-a385-4afd-aed8-91259a20574f/volumes"
Oct 30 22:26:37.790025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:37.790009    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d142ab9-3676-4a91-a470-de5c79226d33" path="/var/lib/kubelet/pods/3d142ab9-3676-4a91-a470-de5c79226d33/volumes"
Oct 30 22:26:38.058631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:38.058541    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:38.751047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:38.751010    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:26:39.340441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.336722    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267" podUID="d5c7c164-2675-4180-b20e-87aa65686a4d" containerName="payload-pre-processing" containerID="cri-o://6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912" gracePeriod=25
Oct 30 22:26:39.596744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.596683    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:26:39.694298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.694266    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c7c164-2675-4180-b20e-87aa65686a4d-kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/d5c7c164-2675-4180-b20e-87aa65686a4d-kube-api-access-h5zm7\") pod \"d5c7c164-2675-4180-b20e-87aa65686a4d\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") "
Oct 30 22:26:39.694500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.694312    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c7c164-2675-4180-b20e-87aa65686a4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c7c164-2675-4180-b20e-87aa65686a4d-plugins-config-volume\") pod \"d5c7c164-2675-4180-b20e-87aa65686a4d\" (UID: \"d5c7c164-2675-4180-b20e-87aa65686a4d\") "
Oct 30 22:26:39.694731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.694704    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c7c164-2675-4180-b20e-87aa65686a4d-plugins-config-volume" pod "d5c7c164-2675-4180-b20e-87aa65686a4d" (UID: "d5c7c164-2675-4180-b20e-87aa65686a4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:39.696484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.696463    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c7c164-2675-4180-b20e-87aa65686a4d-kube-api-access-h5zm7" pod "d5c7c164-2675-4180-b20e-87aa65686a4d" (UID: "d5c7c164-2675-4180-b20e-87aa65686a4d"). InnerVolumeSpecName "kube-api-access-h5zm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:39.795955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.795907    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/d5c7c164-2675-4180-b20e-87aa65686a4d-kube-api-access-h5zm7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:39.795955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:39.795951    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c7c164-2675-4180-b20e-87aa65686a4d-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:40.089484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.089449    5503 generic.go:356] "Generic (PLEG): container finished" podID="d5c7c164-2675-4180-b20e-87aa65686a4d" containerID="6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912" exitCode=0
Oct 30 22:26:40.089650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.089499    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"
Oct 30 22:26:40.089650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.089534    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267" event={"ID":"d5c7c164-2675-4180-b20e-87aa65686a4d","Type":"ContainerDied","Data":"6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912"}
Oct 30 22:26:40.089650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.089588    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267" event={"ID":"d5c7c164-2675-4180-b20e-87aa65686a4d","Type":"ContainerDied","Data":"dbcabd0bd5e0b116d08ccd6b2c32bb33be193c3cafa5eba92d10e55db6124336"}
Oct 30 22:26:40.089650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.089611    5503 scope.go:122] "RemoveContainer" containerID="6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912"
Oct 30 22:26:40.098850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.098767    5503 scope.go:122] "RemoveContainer" containerID="6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912"
Oct 30 22:26:40.099065 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:40.099045    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912\": container with ID starting with 6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912 not found: ID does not exist" containerID="6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912"
Oct 30 22:26:40.099124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.099071    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912"} err="failed to get container status \"6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912\": rpc error: code = NotFound desc = could not find container \"6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912\": container with ID starting with 6c720336335269da042c8f2b2b34e918fd415beeb1b5e4ebbd15a3afbae97912 not found: ID does not exist"
Oct 30 22:26:40.259849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.259820    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"]
Oct 30 22:26:40.310306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:40.310279    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-923d78-74b6b4f5bc-4n267"]
Oct 30 22:26:41.110411 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:41.110376    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:41.790289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:41.790257    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c7c164-2675-4180-b20e-87aa65686a4d" path="/var/lib/kubelet/pods/d5c7c164-2675-4180-b20e-87aa65686a4d/volumes"
Oct 30 22:26:42.317388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:42.317338    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:44.520989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:44.520956    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:44.740497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:44.740466    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:44.971417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:44.971382    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"]
Oct 30 22:26:44.971737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:44.971680    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy" containerID="cri-o://b552ef0b0a57b3cd4caba5c648986d7d4b4eed4eeea65ebc22dc59658d086855" gracePeriod=30
Oct 30 22:26:44.981593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:44.981568    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:45.820612 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.820575    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"]
Oct 30 22:26:45.821239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821218    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d5c7c164-2675-4180-b20e-87aa65686a4d" containerName="payload-pre-processing"
Oct 30 22:26:45.821339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821240    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c7c164-2675-4180-b20e-87aa65686a4d" containerName="payload-pre-processing"
Oct 30 22:26:45.821339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821289    5503 cpu_manager.go:414] "containerMap: removing container" podUID="1024f0de-a385-4afd-aed8-91259a20574f" containerName="main"
Oct 30 22:26:45.821339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821305    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1024f0de-a385-4afd-aed8-91259a20574f" containerName="main"
Oct 30 22:26:45.821339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821317    5503 cpu_manager.go:414] "containerMap: removing container" podUID="1fa96caa-74e1-45e1-a698-3fb9f5631a54" containerName="revoke-keys"
Oct 30 22:26:45.821339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821326    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa96caa-74e1-45e1-a698-3fb9f5631a54" containerName="revoke-keys"
Oct 30 22:26:45.821605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821468    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c7c164-2675-4180-b20e-87aa65686a4d" containerName="payload-pre-processing"
Oct 30 22:26:45.821605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821488    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1024f0de-a385-4afd-aed8-91259a20574f" containerName="main"
Oct 30 22:26:45.821605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.821497    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa96caa-74e1-45e1-a698-3fb9f5631a54" containerName="revoke-keys"
Oct 30 22:26:45.826567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.826542    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.879358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.879320    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.879530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.879382    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qh2v\" (UniqueName: \"kubernetes.io/projected/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-kube-api-access-9qh2v\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.879591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.879567    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.881227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.881205    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:45.882615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.882590    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"]
Oct 30 22:26:45.930058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.930024    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:45.937718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.937689    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:45.937853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.937738    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:26:45.980951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.980920    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.981100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.981049    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.981100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.981078    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qh2v\" (UniqueName: \"kubernetes.io/projected/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-kube-api-access-9qh2v\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.981772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.981754    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:45.983207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:45.983184    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:46.043943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:46.043909    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qh2v\" (UniqueName: \"kubernetes.io/projected/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-kube-api-access-9qh2v\") pod \"maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:46.142655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:46.142556    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:46.232072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:46.232042    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:46.544664 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:26:46.544591    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a596bd9_4a07_48c8_8090_93f79dc6c2c8.slice/crio-dc88d8bf41e57852b690750cc342a6ee68dfb1e274baddad3d10d5fc5c80cf25 WatchSource:0}: Error finding container dc88d8bf41e57852b690750cc342a6ee68dfb1e274baddad3d10d5fc5c80cf25: Status 404 returned error can't find the container with id dc88d8bf41e57852b690750cc342a6ee68dfb1e274baddad3d10d5fc5c80cf25
Oct 30 22:26:46.547030 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:46.547002    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"]
Oct 30 22:26:47.068606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:47.068570    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:47.143551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:47.143472    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t" event={"ID":"4a596bd9-4a07-48c8-8090-93f79dc6c2c8","Type":"ContainerStarted","Data":"dbc1000a1bd7f31bd11a0259d035a0f782a91b1b66e3bf26a7928e67769fd85b"}
Oct 30 22:26:47.143551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:47.143522    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t" event={"ID":"4a596bd9-4a07-48c8-8090-93f79dc6c2c8","Type":"ContainerStarted","Data":"dc88d8bf41e57852b690750cc342a6ee68dfb1e274baddad3d10d5fc5c80cf25"}
Oct 30 22:26:47.268315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:47.268264    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t" podStartSLOduration=2.268250182 podStartE2EDuration="2.268250182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:47.267598152 +0000 UTC m=+2383.602174921" watchObservedRunningTime="2026-10-30 22:26:47.268250182 +0000 UTC m=+2383.602826951"
Oct 30 22:26:47.688498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:47.688421    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:48.151186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:48.151155    5503 generic.go:356] "Generic (PLEG): container finished" podID="4a596bd9-4a07-48c8-8090-93f79dc6c2c8" containerID="dbc1000a1bd7f31bd11a0259d035a0f782a91b1b66e3bf26a7928e67769fd85b" exitCode=0
Oct 30 22:26:48.151628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:48.151237    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t" event={"ID":"4a596bd9-4a07-48c8-8090-93f79dc6c2c8","Type":"ContainerDied","Data":"dbc1000a1bd7f31bd11a0259d035a0f782a91b1b66e3bf26a7928e67769fd85b"}
Oct 30 22:26:49.299517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.299492    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:49.459865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.459794    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-openshift-ca\") pod \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") "
Oct 30 22:26:49.459865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.459843    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-xks-ca\") pod \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") "
Oct 30 22:26:49.460052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.459911    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-kube-api-access-9qh2v\" (UniqueName: \"kubernetes.io/projected/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-kube-api-access-9qh2v\") pod \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\" (UID: \"4a596bd9-4a07-48c8-8090-93f79dc6c2c8\") "
Oct 30 22:26:49.460137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.460115    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-openshift-ca" pod "4a596bd9-4a07-48c8-8090-93f79dc6c2c8" (UID: "4a596bd9-4a07-48c8-8090-93f79dc6c2c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:49.460234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.460220    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:49.462035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.462016    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-kube-api-access-9qh2v" pod "4a596bd9-4a07-48c8-8090-93f79dc6c2c8" (UID: "4a596bd9-4a07-48c8-8090-93f79dc6c2c8"). InnerVolumeSpecName "kube-api-access-9qh2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:49.462118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.462037    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-xks-ca" pod "4a596bd9-4a07-48c8-8090-93f79dc6c2c8" (UID: "4a596bd9-4a07-48c8-8090-93f79dc6c2c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:49.540110 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:49.540079    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:49.561722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.561693    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:49.561722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:49.561717    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qh2v\" (UniqueName: \"kubernetes.io/projected/4a596bd9-4a07-48c8-8090-93f79dc6c2c8-kube-api-access-9qh2v\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.179172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.179137    5503 generic.go:356] "Generic (PLEG): container finished" podID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerID="b552ef0b0a57b3cd4caba5c648986d7d4b4eed4eeea65ebc22dc59658d086855" exitCode=0
Oct 30 22:26:50.179363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.179210    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" event={"ID":"9ea557cd-e74e-4797-bdbc-bc338f21e2e1","Type":"ContainerDied","Data":"b552ef0b0a57b3cd4caba5c648986d7d4b4eed4eeea65ebc22dc59658d086855"}
Oct 30 22:26:50.179363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.179309    5503 scope.go:122] "RemoveContainer" containerID="c0d61e2b19da6d43e5555d3678631cbf9e39946d954034a83faca9ac18ab8245"
Oct 30 22:26:50.180982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.180958    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t" event={"ID":"4a596bd9-4a07-48c8-8090-93f79dc6c2c8","Type":"ContainerDied","Data":"dc88d8bf41e57852b690750cc342a6ee68dfb1e274baddad3d10d5fc5c80cf25"}
Oct 30 22:26:50.181112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.180998    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc88d8bf41e57852b690750cc342a6ee68dfb1e274baddad3d10d5fc5c80cf25"
Oct 30 22:26:50.181112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.181055    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"
Oct 30 22:26:50.219607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.219585    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:26:50.371275 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371192    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-kube-api-access-jd4wf\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371275 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371244    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-podinfo\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371314    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-socket\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371347    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-data\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371392    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-certs\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371441    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-credential-socket\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371508    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istiod-ca-cert\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371595    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-envoy\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371703    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-token\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-token\") pod \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\" (UID: \"9ea557cd-e74e-4797-bdbc-bc338f21e2e1\") "
Oct 30 22:26:50.371803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371749    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-data" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.372232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.371910    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-socket" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.372232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.372044    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-credential-socket" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.372232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.372182    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istiod-ca-cert" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:50.372343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.372325    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.372388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.372345    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.372388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.372362    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.372388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.372377    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.372896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.372862    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-certs" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.373526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.373497    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-podinfo" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:50.373764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.373742    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-kube-api-access-jd4wf" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "kube-api-access-jd4wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:50.374208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.374189    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-token" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:50.374373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.374350    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-envoy" pod "9ea557cd-e74e-4797-bdbc-bc338f21e2e1" (UID: "9ea557cd-e74e-4797-bdbc-bc338f21e2e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.473066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.473034    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.473066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.473064    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.473256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.473075    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-kube-api-access-jd4wf\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.473256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.473084    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.473256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:50.473093    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea557cd-e74e-4797-bdbc-bc338f21e2e1-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:51.122732 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:51.122695    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:26:51.188423 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:51.188396    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"
Oct 30 22:26:51.188608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:51.188446    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95" event={"ID":"9ea557cd-e74e-4797-bdbc-bc338f21e2e1","Type":"ContainerDied","Data":"fef32f6a72d53bfadd6b672d96be2b325eb9e63cd6cc1aa46b5bba2b8d26c565"}
Oct 30 22:26:51.188608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:51.188501    5503 scope.go:122] "RemoveContainer" containerID="b552ef0b0a57b3cd4caba5c648986d7d4b4eed4eeea65ebc22dc59658d086855"
Oct 30 22:26:51.411881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:51.411852    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"]
Oct 30 22:26:51.462419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:51.462387    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-872f4a-openshift-default-c5f469f79-6rw95"]
Oct 30 22:26:51.790343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:51.790270    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" path="/var/lib/kubelet/pods/9ea557cd-e74e-4797-bdbc-bc338f21e2e1/volumes"
Oct 30 22:26:51.916556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:51.916522    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:52.182598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:52.182564    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:53.923576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:53.923533    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:54.076764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:54.076710    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:54.281854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:54.281767    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:55.100448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.100416    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"]
Oct 30 22:26:55.101023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101005    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy"
Oct 30 22:26:55.101080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101027    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy"
Oct 30 22:26:55.101080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101049    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4a596bd9-4a07-48c8-8090-93f79dc6c2c8" containerName="revoke-keys"
Oct 30 22:26:55.101080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101058    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a596bd9-4a07-48c8-8090-93f79dc6c2c8" containerName="revoke-keys"
Oct 30 22:26:55.101176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101079    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy"
Oct 30 22:26:55.101176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101089    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy"
Oct 30 22:26:55.101252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101226    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy"
Oct 30 22:26:55.101252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.101238    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a596bd9-4a07-48c8-8090-93f79dc6c2c8" containerName="revoke-keys"
Oct 30 22:26:55.105953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.105938    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.135052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.135025    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.135183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.135077    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.135288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.135268    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxsw\" (UniqueName: \"kubernetes.io/projected/aca7a654-dd09-46ac-9e64-d95b4851e286-kube-api-access-6bxsw\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.158481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.158453    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"]
Oct 30 22:26:55.206731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.206698    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:55.206903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.206742    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:26:55.206903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.206742    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:26:55.206903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.206799    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:26:55.235953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.235930    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxsw\" (UniqueName: \"kubernetes.io/projected/aca7a654-dd09-46ac-9e64-d95b4851e286-kube-api-access-6bxsw\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.236069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.235990    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.236069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.236028    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.236714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.236694    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.238411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.238389    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.298819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.298784    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxsw\" (UniqueName: \"kubernetes.io/projected/aca7a654-dd09-46ac-9e64-d95b4851e286-kube-api-access-6bxsw\") pod \"maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.422473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.422435    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:55.797985 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:26:55.797958    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca7a654_dd09_46ac_9e64_d95b4851e286.slice/crio-940bdac0e0496fd22b8559d36b3de824c7212eff95ab202b64f6c63d4de35376 WatchSource:0}: Error finding container 940bdac0e0496fd22b8559d36b3de824c7212eff95ab202b64f6c63d4de35376: Status 404 returned error can't find the container with id 940bdac0e0496fd22b8559d36b3de824c7212eff95ab202b64f6c63d4de35376
Oct 30 22:26:55.801229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:55.801206    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"]
Oct 30 22:26:56.225107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:56.225074    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk" event={"ID":"aca7a654-dd09-46ac-9e64-d95b4851e286","Type":"ContainerStarted","Data":"f2e45e8fad1272e80bf654c643200ff08885634acba67616c65c5925db1386a4"}
Oct 30 22:26:56.225558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:56.225129    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk" event={"ID":"aca7a654-dd09-46ac-9e64-d95b4851e286","Type":"ContainerStarted","Data":"940bdac0e0496fd22b8559d36b3de824c7212eff95ab202b64f6c63d4de35376"}
Oct 30 22:26:56.344419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:56.344370    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk" podStartSLOduration=1.344357254 podStartE2EDuration="1.344357254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:56.342591893 +0000 UTC m=+2392.677168662" watchObservedRunningTime="2026-10-30 22:26:56.344357254 +0000 UTC m=+2392.678934022"
Oct 30 22:26:56.560521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:56.560484    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:57.232130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:57.232093    5503 generic.go:356] "Generic (PLEG): container finished" podID="aca7a654-dd09-46ac-9e64-d95b4851e286" containerID="f2e45e8fad1272e80bf654c643200ff08885634acba67616c65c5925db1386a4" exitCode=0
Oct 30 22:26:57.232520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:57.232145    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk" event={"ID":"aca7a654-dd09-46ac-9e64-d95b4851e286","Type":"ContainerDied","Data":"f2e45e8fad1272e80bf654c643200ff08885634acba67616c65c5925db1386a4"}
Oct 30 22:26:58.099653 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:26:58.099612    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:26:58.106858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.106829    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:58.368115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.368087    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:58.387489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.387463    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:58.477079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.477028    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-openshift-ca\") pod \"aca7a654-dd09-46ac-9e64-d95b4851e286\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") "
Oct 30 22:26:58.477279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.477213    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca7a654-dd09-46ac-9e64-d95b4851e286-kube-api-access-6bxsw\" (UniqueName: \"kubernetes.io/projected/aca7a654-dd09-46ac-9e64-d95b4851e286-kube-api-access-6bxsw\") pod \"aca7a654-dd09-46ac-9e64-d95b4851e286\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") "
Oct 30 22:26:58.477364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.477354    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-xks-ca\") pod \"aca7a654-dd09-46ac-9e64-d95b4851e286\" (UID: \"aca7a654-dd09-46ac-9e64-d95b4851e286\") "
Oct 30 22:26:58.481370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.481322    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-openshift-ca" pod "aca7a654-dd09-46ac-9e64-d95b4851e286" (UID: "aca7a654-dd09-46ac-9e64-d95b4851e286"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:58.484556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.483468    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-xks-ca" pod "aca7a654-dd09-46ac-9e64-d95b4851e286" (UID: "aca7a654-dd09-46ac-9e64-d95b4851e286"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:58.484556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.483748    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca7a654-dd09-46ac-9e64-d95b4851e286-kube-api-access-6bxsw" pod "aca7a654-dd09-46ac-9e64-d95b4851e286" (UID: "aca7a654-dd09-46ac-9e64-d95b4851e286"). InnerVolumeSpecName "kube-api-access-6bxsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:58.521182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.521144    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:58.580193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.580154    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:58.580193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.580190    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxsw\" (UniqueName: \"kubernetes.io/projected/aca7a654-dd09-46ac-9e64-d95b4851e286-kube-api-access-6bxsw\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:58.580446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.580215    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aca7a654-dd09-46ac-9e64-d95b4851e286-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:58.659083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.659053    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"]
Oct 30 22:26:58.663431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:58.663409    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c89e47-rct6t"]
Oct 30 22:26:59.142884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:59.142799    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:26:59.297566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:59.297526    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk" event={"ID":"aca7a654-dd09-46ac-9e64-d95b4851e286","Type":"ContainerDied","Data":"940bdac0e0496fd22b8559d36b3de824c7212eff95ab202b64f6c63d4de35376"}
Oct 30 22:26:59.298082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:59.298059    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="940bdac0e0496fd22b8559d36b3de824c7212eff95ab202b64f6c63d4de35376"
Oct 30 22:26:59.298082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:59.297946    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"
Oct 30 22:26:59.789312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:26:59.789276    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a596bd9-4a07-48c8-8090-93f79dc6c2c8" path="/var/lib/kubelet/pods/4a596bd9-4a07-48c8-8090-93f79dc6c2c8/volumes"
Oct 30 22:27:00.489931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:00.489900    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:00.654129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:00.654091    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:01.189375 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:27:01.189338    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a0393b_7a31_46a5_ab6f_9143f2d72a11.slice/crio-c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:27:02.707301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:02.707267    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:04.122563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:04.122538    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:27:04.162175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:04.162151    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:27:04.215560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:04.215523    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:04.523810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:04.523781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:04.674793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:04.674759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:04.705056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:04.705022    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"]
Oct 30 22:27:04.755910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:04.755878    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ebcc6d-t5nrd"]
Oct 30 22:27:05.159895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:05.159867    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:05.788278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:05.788245    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a0393b-7a31-46a5-ab6f-9143f2d72a11" path="/var/lib/kubelet/pods/b5a0393b-7a31-46a5-ab6f-9143f2d72a11/volumes"
Oct 30 22:27:05.946526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:05.946495    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.101906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:06.101822    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:06.328964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:06.328921    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:07.878247 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:07.878206    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:08.574878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:08.574843    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:08.734553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:08.734514    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:09.504625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:09.504584    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:11.264170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:11.264137    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:11.414702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:11.414665    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:11.627560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:11.627474    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"]
Oct 30 22:27:11.627794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:11.627718    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy" containerID="cri-o://a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5" gracePeriod=30
Oct 30 22:27:11.651089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:11.651044    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.323047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:13.323002    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.495035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:13.494998    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:13.811689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:13.811649    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:15.073811 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:15.073779    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:15.623087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:15.623048    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:15.779367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:15.779327    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.071287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.071252    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.380302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.380223    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.541964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.541931    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.617924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.617891    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"]
Oct 30 22:27:16.723124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.723081    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:16.729074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.729039    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-923d78-2xvdp"]
Oct 30 22:27:16.924526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.924502    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:27:16.972142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972067    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-certs\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972119    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-kube-api-access-2nq2b\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-kube-api-access-2nq2b\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972152    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-credential-socket\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972171    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-podinfo\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972196    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-socket\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972351    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-data\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972403    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-certs" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:16.972492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972480    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istiod-ca-cert\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972498    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-socket" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:16.972854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972513    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-credential-socket" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:16.972854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972525    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-token\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-token\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972594    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-envoy\") pod \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\" (UID: \"3e1fcecc-4346-4ace-89d7-1cd3aecf541b\") "
Oct 30 22:27:16.972854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972725    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-data" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:16.973132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.972887    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istiod-ca-cert" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:16.973132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.973110    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:16.973132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.973131    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:16.973292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.973145    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:16.973292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.973160    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:16.973292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.973171    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:16.974557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.974533    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-kube-api-access-2nq2b" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "kube-api-access-2nq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:16.974720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.974604    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-token" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:16.974866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.974847    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-podinfo" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:16.975011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:16.974865    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-envoy" pod "3e1fcecc-4346-4ace-89d7-1cd3aecf541b" (UID: "3e1fcecc-4346-4ace-89d7-1cd3aecf541b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:17.019357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.019312    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.074453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.074427    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:17.074453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.074452    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:17.074453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.074464    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nq2b\" (UniqueName: \"kubernetes.io/projected/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-kube-api-access-2nq2b\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:17.074717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.074477    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e1fcecc-4346-4ace-89d7-1cd3aecf541b-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:17.204963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.204925    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:17.455538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.455502    5503 generic.go:356] "Generic (PLEG): container finished" podID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerID="a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5" exitCode=0
Oct 30 22:27:17.456074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.455563    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"
Oct 30 22:27:17.456074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.455580    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" event={"ID":"3e1fcecc-4346-4ace-89d7-1cd3aecf541b","Type":"ContainerDied","Data":"a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5"}
Oct 30 22:27:17.456074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.455662    5503 scope.go:122] "RemoveContainer" containerID="a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5"
Oct 30 22:27:17.456074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.455666    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb" event={"ID":"3e1fcecc-4346-4ace-89d7-1cd3aecf541b","Type":"ContainerDied","Data":"70edd46a408cf7f883228bca40a541839855733aff2fed1a2319ea4b2d7d945e"}
Oct 30 22:27:17.468222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.468194    5503 scope.go:122] "RemoveContainer" containerID="5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0"
Oct 30 22:27:17.477088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.477065    5503 scope.go:122] "RemoveContainer" containerID="a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5"
Oct 30 22:27:17.477442 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:27:17.477422    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5\": container with ID starting with a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5 not found: ID does not exist" containerID="a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5"
Oct 30 22:27:17.477521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.477452    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5"} err="failed to get container status \"a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5\": rpc error: code = NotFound desc = could not find container \"a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5\": container with ID starting with a0d7012782b85dc0a66e0bd6e2a2527c88aceade35ae365066cb169c74ec50a5 not found: ID does not exist"
Oct 30 22:27:17.477521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.477474    5503 scope.go:122] "RemoveContainer" containerID="5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0"
Oct 30 22:27:17.477831 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:27:17.477808    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0\": container with ID starting with 5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0 not found: ID does not exist" containerID="5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0"
Oct 30 22:27:17.477927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.477833    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0"} err="failed to get container status \"5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0\": rpc error: code = NotFound desc = could not find container \"5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0\": container with ID starting with 5855789ad42161caf6b20804671d75025b4df077c6a2e2cdc1e5824c304a93e0 not found: ID does not exist"
Oct 30 22:27:17.629199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.629162    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"]
Oct 30 22:27:17.634795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.634769    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ebcc6d-openshift-default-58cb964577-ltvvb"]
Oct 30 22:27:17.788436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.788363    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa96caa-74e1-45e1-a698-3fb9f5631a54" path="/var/lib/kubelet/pods/1fa96caa-74e1-45e1-a698-3fb9f5631a54/volumes"
Oct 30 22:27:17.788873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.788858    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" path="/var/lib/kubelet/pods/3e1fcecc-4346-4ace-89d7-1cd3aecf541b/volumes"
Oct 30 22:27:17.818582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:17.818555    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:18.133988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:18.133917    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:18.371813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:18.371767    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"]
Oct 30 22:27:18.376750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:18.376717    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6a3357-lh5vk"]
Oct 30 22:27:18.867654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:18.867610    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:19.776228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776194    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"]
Oct 30 22:27:19.776712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776698    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy"
Oct 30 22:27:19.776772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776714    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy"
Oct 30 22:27:19.776772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776766    5503 cpu_manager.go:414] "containerMap: removing container" podUID="aca7a654-dd09-46ac-9e64-d95b4851e286" containerName="revoke-keys"
Oct 30 22:27:19.776842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776773    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca7a654-dd09-46ac-9e64-d95b4851e286" containerName="revoke-keys"
Oct 30 22:27:19.776874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776847    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca7a654-dd09-46ac-9e64-d95b4851e286" containerName="revoke-keys"
Oct 30 22:27:19.776874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776859    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy"
Oct 30 22:27:19.776874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776866    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy"
Oct 30 22:27:19.776963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.776877    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ea557cd-e74e-4797-bdbc-bc338f21e2e1" containerName="istio-proxy"
Oct 30 22:27:19.781854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.781836    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.789882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.789856    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca7a654-dd09-46ac-9e64-d95b4851e286" path="/var/lib/kubelet/pods/aca7a654-dd09-46ac-9e64-d95b4851e286/volumes"
Oct 30 22:27:19.812908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.812878    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.813104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.813084    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.813153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.813130    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnxf\" (UniqueName: \"kubernetes.io/projected/317e9b4e-16f5-43ef-96e5-2f5b427937c3-kube-api-access-ghnxf\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.833982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.833950    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"]
Oct 30 22:27:19.882790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.882759    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:19.882790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.882766    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:19.882790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.882781    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:19.887476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.887456    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:27:19.913885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.913855    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.914084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.914010    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.914084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.914048    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnxf\" (UniqueName: \"kubernetes.io/projected/317e9b4e-16f5-43ef-96e5-2f5b427937c3-kube-api-access-ghnxf\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.914666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.914626    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.916169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.916148    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:19.980016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:19.979976    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnxf\" (UniqueName: \"kubernetes.io/projected/317e9b4e-16f5-43ef-96e5-2f5b427937c3-kube-api-access-ghnxf\") pod \"maas-api-revoke-sub-06f2db6d-7xknd\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:20.096414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:20.096335    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:20.464662 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:27:20.464596    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317e9b4e_16f5_43ef_96e5_2f5b427937c3.slice/crio-5cc766aefaafc852a670718ece19ea62b7505af35731f6bb18ba172900c24ddc WatchSource:0}: Error finding container 5cc766aefaafc852a670718ece19ea62b7505af35731f6bb18ba172900c24ddc: Status 404 returned error can't find the container with id 5cc766aefaafc852a670718ece19ea62b7505af35731f6bb18ba172900c24ddc
Oct 30 22:27:20.466776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:20.466756    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"]
Oct 30 22:27:20.475109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:20.475080    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd" event={"ID":"317e9b4e-16f5-43ef-96e5-2f5b427937c3","Type":"ContainerStarted","Data":"5cc766aefaafc852a670718ece19ea62b7505af35731f6bb18ba172900c24ddc"}
Oct 30 22:27:21.483071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:21.483024    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd" event={"ID":"317e9b4e-16f5-43ef-96e5-2f5b427937c3","Type":"ContainerStarted","Data":"a68f0a4bf85505642009339f65ceec934940be64161e5a9db13ffb6c05bffe17"}
Oct 30 22:27:21.604369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:21.604317    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd" podStartSLOduration=2.604304703 podStartE2EDuration="2.604304703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:21.601756788 +0000 UTC m=+2417.936333557" watchObservedRunningTime="2026-10-30 22:27:21.604304703 +0000 UTC m=+2417.938881472"
Oct 30 22:27:22.100619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:22.100586    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:22.490721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:22.490686    5503 generic.go:356] "Generic (PLEG): container finished" podID="317e9b4e-16f5-43ef-96e5-2f5b427937c3" containerID="a68f0a4bf85505642009339f65ceec934940be64161e5a9db13ffb6c05bffe17" exitCode=0
Oct 30 22:27:22.491140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:22.490774    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd" event={"ID":"317e9b4e-16f5-43ef-96e5-2f5b427937c3","Type":"ContainerDied","Data":"a68f0a4bf85505642009339f65ceec934940be64161e5a9db13ffb6c05bffe17"}
Oct 30 22:27:23.019945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.019893    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:23.185586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.185556    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:23.362215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.362105    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:23.521747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.521705    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:23.632019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.631959    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:23.658471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.658445    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317e9b4e-16f5-43ef-96e5-2f5b427937c3-kube-api-access-ghnxf\" (UniqueName: \"kubernetes.io/projected/317e9b4e-16f5-43ef-96e5-2f5b427937c3-kube-api-access-ghnxf\") pod \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") "
Oct 30 22:27:23.658629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.658517    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-openshift-ca\") pod \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") "
Oct 30 22:27:23.658629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.658571    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-xks-ca\") pod \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\" (UID: \"317e9b4e-16f5-43ef-96e5-2f5b427937c3\") "
Oct 30 22:27:23.658904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.658873    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-openshift-ca" pod "317e9b4e-16f5-43ef-96e5-2f5b427937c3" (UID: "317e9b4e-16f5-43ef-96e5-2f5b427937c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:23.660560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.660541    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-xks-ca" pod "317e9b4e-16f5-43ef-96e5-2f5b427937c3" (UID: "317e9b4e-16f5-43ef-96e5-2f5b427937c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:23.660703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.660683    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317e9b4e-16f5-43ef-96e5-2f5b427937c3-kube-api-access-ghnxf" pod "317e9b4e-16f5-43ef-96e5-2f5b427937c3" (UID: "317e9b4e-16f5-43ef-96e5-2f5b427937c3"). InnerVolumeSpecName "kube-api-access-ghnxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:23.760477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.760445    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnxf\" (UniqueName: \"kubernetes.io/projected/317e9b4e-16f5-43ef-96e5-2f5b427937c3-kube-api-access-ghnxf\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:23.760477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.760474    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:23.760477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:23.760486    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317e9b4e-16f5-43ef-96e5-2f5b427937c3-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:24.510961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:24.510927    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"
Oct 30 22:27:24.511137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:24.510960    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd" event={"ID":"317e9b4e-16f5-43ef-96e5-2f5b427937c3","Type":"ContainerDied","Data":"5cc766aefaafc852a670718ece19ea62b7505af35731f6bb18ba172900c24ddc"}
Oct 30 22:27:24.511137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:24.511047    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cc766aefaafc852a670718ece19ea62b7505af35731f6bb18ba172900c24ddc"
Oct 30 22:27:24.548889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:24.548854    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:24.927356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:24.927319    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:25.078695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:25.078660    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:25.292345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:25.292262    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:26.080622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:26.080587    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:27:26.870775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:26.870743    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:27.037903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:27.037865    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:27.559089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:27.559056    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:27.874996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:27.874897    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:28.100777 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:27:28.100743    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:27:31.273440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.273403    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"]
Oct 30 22:27:31.273982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.273862    5503 cpu_manager.go:414] "containerMap: removing container" podUID="317e9b4e-16f5-43ef-96e5-2f5b427937c3" containerName="revoke-keys"
Oct 30 22:27:31.273982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.273877    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317e9b4e-16f5-43ef-96e5-2f5b427937c3" containerName="revoke-keys"
Oct 30 22:27:31.273982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.273891    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy"
Oct 30 22:27:31.273982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.273898    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1fcecc-4346-4ace-89d7-1cd3aecf541b" containerName="istio-proxy"
Oct 30 22:27:31.274204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.274010    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="317e9b4e-16f5-43ef-96e5-2f5b427937c3" containerName="revoke-keys"
Oct 30 22:27:31.277238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.277220    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.339736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.339705    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:31.342083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.342057    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"]
Oct 30 22:27:31.373708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.373678    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.373855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.373768    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5rpg\" (UniqueName: \"kubernetes.io/projected/993c87b6-f419-413c-9e41-f8585635cfee-kube-api-access-s5rpg\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.373975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.373953    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.390555 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.390531    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:31.390555 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.390533    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:31.390765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.390570    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:27:31.429871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.429843    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:31.474755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.474724    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.474897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.474774    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.474897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.474834    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5rpg\" (UniqueName: \"kubernetes.io/projected/993c87b6-f419-413c-9e41-f8585635cfee-kube-api-access-s5rpg\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.475427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.475402    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.477128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.477107    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.540047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.539978    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5rpg\" (UniqueName: \"kubernetes.io/projected/993c87b6-f419-413c-9e41-f8585635cfee-kube-api-access-s5rpg\") pod \"maas-api-revoke-sub-6d9e6a1a-sbfgm\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.591674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.591629    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:31.599505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.599477    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:31.757241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.757208    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:31.911897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.911870    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:31.974246 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:27:31.974215    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993c87b6_f419_413c_9e41_f8585635cfee.slice/crio-e8e3685fbb03615ba3697e7746922f6ce802ba00c0b15bd6c0495f3fbc93d51b WatchSource:0}: Error finding container e8e3685fbb03615ba3697e7746922f6ce802ba00c0b15bd6c0495f3fbc93d51b: Status 404 returned error can't find the container with id e8e3685fbb03615ba3697e7746922f6ce802ba00c0b15bd6c0495f3fbc93d51b
Oct 30 22:27:31.977371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:31.977347    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"]
Oct 30 22:27:32.588222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:32.588137    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm" event={"ID":"993c87b6-f419-413c-9e41-f8585635cfee","Type":"ContainerStarted","Data":"580ff4cd10fa475089684e7eceee4d58bdf5e2b6818a6bcc2a069f4c1049a0a1"}
Oct 30 22:27:32.588222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:32.588191    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm" event={"ID":"993c87b6-f419-413c-9e41-f8585635cfee","Type":"ContainerStarted","Data":"e8e3685fbb03615ba3697e7746922f6ce802ba00c0b15bd6c0495f3fbc93d51b"}
Oct 30 22:27:32.710306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:32.710244    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm" podStartSLOduration=1.710228917 podStartE2EDuration="1.710228917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:32.706239982 +0000 UTC m=+2429.040816750" watchObservedRunningTime="2026-10-30 22:27:32.710228917 +0000 UTC m=+2429.044805685"
Oct 30 22:27:33.595556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:33.595520    5503 generic.go:356] "Generic (PLEG): container finished" podID="993c87b6-f419-413c-9e41-f8585635cfee" containerID="580ff4cd10fa475089684e7eceee4d58bdf5e2b6818a6bcc2a069f4c1049a0a1" exitCode=0
Oct 30 22:27:33.595967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:33.595610    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm" event={"ID":"993c87b6-f419-413c-9e41-f8585635cfee","Type":"ContainerDied","Data":"580ff4cd10fa475089684e7eceee4d58bdf5e2b6818a6bcc2a069f4c1049a0a1"}
Oct 30 22:27:34.751007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.750982    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:34.817374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.817346    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-openshift-ca\") pod \"993c87b6-f419-413c-9e41-f8585635cfee\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") "
Oct 30 22:27:34.817561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.817544    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993c87b6-f419-413c-9e41-f8585635cfee-kube-api-access-s5rpg\" (UniqueName: \"kubernetes.io/projected/993c87b6-f419-413c-9e41-f8585635cfee-kube-api-access-s5rpg\") pod \"993c87b6-f419-413c-9e41-f8585635cfee\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") "
Oct 30 22:27:34.817605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.817590    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-xks-ca\") pod \"993c87b6-f419-413c-9e41-f8585635cfee\" (UID: \"993c87b6-f419-413c-9e41-f8585635cfee\") "
Oct 30 22:27:34.817776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.817755    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-openshift-ca" pod "993c87b6-f419-413c-9e41-f8585635cfee" (UID: "993c87b6-f419-413c-9e41-f8585635cfee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:34.818013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.817991    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:34.819609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.819583    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-xks-ca" pod "993c87b6-f419-413c-9e41-f8585635cfee" (UID: "993c87b6-f419-413c-9e41-f8585635cfee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:34.819754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.819609    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993c87b6-f419-413c-9e41-f8585635cfee-kube-api-access-s5rpg" pod "993c87b6-f419-413c-9e41-f8585635cfee" (UID: "993c87b6-f419-413c-9e41-f8585635cfee"). InnerVolumeSpecName "kube-api-access-s5rpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:34.918833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.918797    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5rpg\" (UniqueName: \"kubernetes.io/projected/993c87b6-f419-413c-9e41-f8585635cfee-kube-api-access-s5rpg\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:34.918833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:34.918827    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993c87b6-f419-413c-9e41-f8585635cfee-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:35.611229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:35.611201    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"
Oct 30 22:27:35.611422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:35.611236    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm" event={"ID":"993c87b6-f419-413c-9e41-f8585635cfee","Type":"ContainerDied","Data":"e8e3685fbb03615ba3697e7746922f6ce802ba00c0b15bd6c0495f3fbc93d51b"}
Oct 30 22:27:35.611422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:35.611284    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8e3685fbb03615ba3697e7746922f6ce802ba00c0b15bd6c0495f3fbc93d51b"
Oct 30 22:27:36.018308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.018272    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:36.219476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.219445    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:36.633826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.633794    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:36.759684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.759631    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"]
Oct 30 22:27:36.760309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.760287    5503 cpu_manager.go:414] "containerMap: removing container" podUID="993c87b6-f419-413c-9e41-f8585635cfee" containerName="revoke-keys"
Oct 30 22:27:36.760309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.760309    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993c87b6-f419-413c-9e41-f8585635cfee" containerName="revoke-keys"
Oct 30 22:27:36.760445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.760420    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="993c87b6-f419-413c-9e41-f8585635cfee" containerName="revoke-keys"
Oct 30 22:27:36.763495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.763468    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.818815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.818773    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:36.819006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.818773    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:36.819061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.818957    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"]
Oct 30 22:27:36.850085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.850046    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.850334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.850309    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pg8\" (UniqueName: \"kubernetes.io/projected/7676afea-3648-4d80-81c8-afbdb480adf7-kube-api-access-j9pg8\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.850412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.850369    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.868043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.868012    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:27:36.868238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.868066    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:36.951323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.951287    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pg8\" (UniqueName: \"kubernetes.io/projected/7676afea-3648-4d80-81c8-afbdb480adf7-kube-api-access-j9pg8\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.951509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.951340    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.951509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.951388    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.952068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.952039    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:36.953801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:36.953777    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:37.014106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:37.014072    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pg8\" (UniqueName: \"kubernetes.io/projected/7676afea-3648-4d80-81c8-afbdb480adf7-kube-api-access-j9pg8\") pod \"maas-api-revoke-sub-f5065faa-lt6v4\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:37.081183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:37.081143    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:37.451622 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:27:37.451591    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7676afea_3648_4d80_81c8_afbdb480adf7.slice/crio-68b4957fc3f329f19b9ce9115df90e24d82bcb7531c7394d26d69b03b8e96e06 WatchSource:0}: Error finding container 68b4957fc3f329f19b9ce9115df90e24d82bcb7531c7394d26d69b03b8e96e06: Status 404 returned error can't find the container with id 68b4957fc3f329f19b9ce9115df90e24d82bcb7531c7394d26d69b03b8e96e06
Oct 30 22:27:37.454970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:37.454940    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"]
Oct 30 22:27:37.634818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:37.634774    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4" event={"ID":"7676afea-3648-4d80-81c8-afbdb480adf7","Type":"ContainerStarted","Data":"68b4957fc3f329f19b9ce9115df90e24d82bcb7531c7394d26d69b03b8e96e06"}
Oct 30 22:27:38.642201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:38.642117    5503 generic.go:356] "Generic (PLEG): container finished" podID="7676afea-3648-4d80-81c8-afbdb480adf7" containerID="ab10c15362ff88fafa3d3938650cf9f7d304f373da11ee61ed3582489ea4684c" exitCode=0
Oct 30 22:27:38.642566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:38.642196    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4" event={"ID":"7676afea-3648-4d80-81c8-afbdb480adf7","Type":"ContainerDied","Data":"ab10c15362ff88fafa3d3938650cf9f7d304f373da11ee61ed3582489ea4684c"}
Oct 30 22:27:39.779943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.779915    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:39.882462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.882427    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-xks-ca\") pod \"7676afea-3648-4d80-81c8-afbdb480adf7\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") "
Oct 30 22:27:39.882678 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.882473    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7676afea-3648-4d80-81c8-afbdb480adf7-kube-api-access-j9pg8\" (UniqueName: \"kubernetes.io/projected/7676afea-3648-4d80-81c8-afbdb480adf7-kube-api-access-j9pg8\") pod \"7676afea-3648-4d80-81c8-afbdb480adf7\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") "
Oct 30 22:27:39.882678 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.882600    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-openshift-ca\") pod \"7676afea-3648-4d80-81c8-afbdb480adf7\" (UID: \"7676afea-3648-4d80-81c8-afbdb480adf7\") "
Oct 30 22:27:39.882970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.882944    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-openshift-ca" pod "7676afea-3648-4d80-81c8-afbdb480adf7" (UID: "7676afea-3648-4d80-81c8-afbdb480adf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:39.883044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.883031    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:39.884589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.884566    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-xks-ca" pod "7676afea-3648-4d80-81c8-afbdb480adf7" (UID: "7676afea-3648-4d80-81c8-afbdb480adf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:39.884671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.884629    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7676afea-3648-4d80-81c8-afbdb480adf7-kube-api-access-j9pg8" pod "7676afea-3648-4d80-81c8-afbdb480adf7" (UID: "7676afea-3648-4d80-81c8-afbdb480adf7"). InnerVolumeSpecName "kube-api-access-j9pg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:39.984107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.984072    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7676afea-3648-4d80-81c8-afbdb480adf7-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:39.984107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:39.984103    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pg8\" (UniqueName: \"kubernetes.io/projected/7676afea-3648-4d80-81c8-afbdb480adf7-kube-api-access-j9pg8\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:40.656756 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:40.656727    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"
Oct 30 22:27:40.656939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:40.656728    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4" event={"ID":"7676afea-3648-4d80-81c8-afbdb480adf7","Type":"ContainerDied","Data":"68b4957fc3f329f19b9ce9115df90e24d82bcb7531c7394d26d69b03b8e96e06"}
Oct 30 22:27:40.656939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:40.656855    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68b4957fc3f329f19b9ce9115df90e24d82bcb7531c7394d26d69b03b8e96e06"
Oct 30 22:27:41.337749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:41.337713    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:42.081739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:42.081702    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:45.102748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.102717    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"]
Oct 30 22:27:45.103194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.103181    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7676afea-3648-4d80-81c8-afbdb480adf7" containerName="revoke-keys"
Oct 30 22:27:45.103239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.103194    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7676afea-3648-4d80-81c8-afbdb480adf7" containerName="revoke-keys"
Oct 30 22:27:45.103305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.103296    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7676afea-3648-4d80-81c8-afbdb480adf7" containerName="revoke-keys"
Oct 30 22:27:45.106075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.106057    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.166880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.166848    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"]
Oct 30 22:27:45.215318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.215284    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:45.215489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.215319    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:27:45.215489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.215364    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:27:45.215489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.215371    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:27:45.250038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.250003    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.250211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.250190    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.250257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.250219    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldpz\" (UniqueName: \"kubernetes.io/projected/762d2b70-cf93-4379-a99a-906167aba75b-kube-api-access-vldpz\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.351120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.351085    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.351120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.351122    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldpz\" (UniqueName: \"kubernetes.io/projected/762d2b70-cf93-4379-a99a-906167aba75b-kube-api-access-vldpz\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.351355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.351193    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.351900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.351876    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.353505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.353463    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.414265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.414232    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldpz\" (UniqueName: \"kubernetes.io/projected/762d2b70-cf93-4379-a99a-906167aba75b-kube-api-access-vldpz\") pod \"maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.419039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.419017    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:45.789557 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:27:45.789525    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762d2b70_cf93_4379_a99a_906167aba75b.slice/crio-9adbe4675e424a1ce49d7d0d304f1f2d2b078a9297f00f9ba3d592f31e66d21f WatchSource:0}: Error finding container 9adbe4675e424a1ce49d7d0d304f1f2d2b078a9297f00f9ba3d592f31e66d21f: Status 404 returned error can't find the container with id 9adbe4675e424a1ce49d7d0d304f1f2d2b078a9297f00f9ba3d592f31e66d21f
Oct 30 22:27:45.791989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:45.791965    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"]
Oct 30 22:27:46.240969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:46.240931    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:46.694859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:46.694821    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z" event={"ID":"762d2b70-cf93-4379-a99a-906167aba75b","Type":"ContainerStarted","Data":"3f93db148363992da06608bd6144deaaacba213f4b3fd7e34d428643ec6d82b2"}
Oct 30 22:27:46.695045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:46.694883    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z" event={"ID":"762d2b70-cf93-4379-a99a-906167aba75b","Type":"ContainerStarted","Data":"9adbe4675e424a1ce49d7d0d304f1f2d2b078a9297f00f9ba3d592f31e66d21f"}
Oct 30 22:27:46.813868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:46.813814    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z" podStartSLOduration=1.8137996159999998 podStartE2EDuration="1.813799616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:46.813044147 +0000 UTC m=+2443.147620916" watchObservedRunningTime="2026-10-30 22:27:46.813799616 +0000 UTC m=+2443.148376384"
Oct 30 22:27:47.618108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.618073    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"]
Oct 30 22:27:47.622440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.622414    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.680122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.680080    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.680323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.680234    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.680399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.680336    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ghx\" (UniqueName: \"kubernetes.io/projected/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-kube-api-access-v7ghx\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.682457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.682425    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"]
Oct 30 22:27:47.703084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.703051    5503 generic.go:356] "Generic (PLEG): container finished" podID="762d2b70-cf93-4379-a99a-906167aba75b" containerID="3f93db148363992da06608bd6144deaaacba213f4b3fd7e34d428643ec6d82b2" exitCode=0
Oct 30 22:27:47.703241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.703142    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z" event={"ID":"762d2b70-cf93-4379-a99a-906167aba75b","Type":"ContainerDied","Data":"3f93db148363992da06608bd6144deaaacba213f4b3fd7e34d428643ec6d82b2"}
Oct 30 22:27:47.781342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.781299    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ghx\" (UniqueName: \"kubernetes.io/projected/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-kube-api-access-v7ghx\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.781605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.781425    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.781605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.781476    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.782190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.782163    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.783959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.783934    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.845353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.845307    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ghx\" (UniqueName: \"kubernetes.io/projected/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-kube-api-access-v7ghx\") pod \"maas-api-revoke-sub-3c3a6ba9-t4c7q\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:47.943576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:47.943545    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:48.321159 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:27:48.321117    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f5dfcb_f6f1_4f5f_874a_a15df43fe8c8.slice/crio-d5a70d1b0e258f51aece9330f918c54cd33985a6da69550ce0669c60de3e6afb WatchSource:0}: Error finding container d5a70d1b0e258f51aece9330f918c54cd33985a6da69550ce0669c60de3e6afb: Status 404 returned error can't find the container with id d5a70d1b0e258f51aece9330f918c54cd33985a6da69550ce0669c60de3e6afb
Oct 30 22:27:48.322821 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.322801    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"]
Oct 30 22:27:48.714189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.714154    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q" event={"ID":"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8","Type":"ContainerStarted","Data":"697f743f4ab75d9b332d1088297247848cbd52b47db2eeaf1685c359208f186f"}
Oct 30 22:27:48.714628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.714208    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q" event={"ID":"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8","Type":"ContainerStarted","Data":"d5a70d1b0e258f51aece9330f918c54cd33985a6da69550ce0669c60de3e6afb"}
Oct 30 22:27:48.834863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.834811    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q" podStartSLOduration=1.8347964110000001 podStartE2EDuration="1.834796411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:48.833361409 +0000 UTC m=+2445.167938176" watchObservedRunningTime="2026-10-30 22:27:48.834796411 +0000 UTC m=+2445.169373180"
Oct 30 22:27:48.867058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.867033    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:48.894550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.894471    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-xks-ca\") pod \"762d2b70-cf93-4379-a99a-906167aba75b\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") "
Oct 30 22:27:48.894727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.894712    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-openshift-ca\") pod \"762d2b70-cf93-4379-a99a-906167aba75b\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") "
Oct 30 22:27:48.894801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.894784    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762d2b70-cf93-4379-a99a-906167aba75b-kube-api-access-vldpz\" (UniqueName: \"kubernetes.io/projected/762d2b70-cf93-4379-a99a-906167aba75b-kube-api-access-vldpz\") pod \"762d2b70-cf93-4379-a99a-906167aba75b\" (UID: \"762d2b70-cf93-4379-a99a-906167aba75b\") "
Oct 30 22:27:48.895057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.895030    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-openshift-ca" pod "762d2b70-cf93-4379-a99a-906167aba75b" (UID: "762d2b70-cf93-4379-a99a-906167aba75b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:48.895184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.895166    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.896716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.896690    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-xks-ca" pod "762d2b70-cf93-4379-a99a-906167aba75b" (UID: "762d2b70-cf93-4379-a99a-906167aba75b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:48.896817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.896738    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762d2b70-cf93-4379-a99a-906167aba75b-kube-api-access-vldpz" pod "762d2b70-cf93-4379-a99a-906167aba75b" (UID: "762d2b70-cf93-4379-a99a-906167aba75b"). InnerVolumeSpecName "kube-api-access-vldpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:48.997110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.997073    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vldpz\" (UniqueName: \"kubernetes.io/projected/762d2b70-cf93-4379-a99a-906167aba75b-kube-api-access-vldpz\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.997110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:48.997113    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/762d2b70-cf93-4379-a99a-906167aba75b-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:49.721167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:49.721137    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"
Oct 30 22:27:49.721723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:49.721170    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z" event={"ID":"762d2b70-cf93-4379-a99a-906167aba75b","Type":"ContainerDied","Data":"9adbe4675e424a1ce49d7d0d304f1f2d2b078a9297f00f9ba3d592f31e66d21f"}
Oct 30 22:27:49.721723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:49.721228    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9adbe4675e424a1ce49d7d0d304f1f2d2b078a9297f00f9ba3d592f31e66d21f"
Oct 30 22:27:49.722865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:49.722832    5503 generic.go:356] "Generic (PLEG): container finished" podID="e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" containerID="697f743f4ab75d9b332d1088297247848cbd52b47db2eeaf1685c359208f186f" exitCode=0
Oct 30 22:27:49.722998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:49.722906    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q" event={"ID":"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8","Type":"ContainerDied","Data":"697f743f4ab75d9b332d1088297247848cbd52b47db2eeaf1685c359208f186f"}
Oct 30 22:27:50.880040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.880008    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:50.918477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.918442    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-kube-api-access-v7ghx\" (UniqueName: \"kubernetes.io/projected/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-kube-api-access-v7ghx\") pod \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") "
Oct 30 22:27:50.918686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.918579    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-openshift-ca\") pod \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") "
Oct 30 22:27:50.918686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.918600    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-xks-ca\") pod \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\" (UID: \"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8\") "
Oct 30 22:27:50.918960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.918929    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-openshift-ca" pod "e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" (UID: "e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:50.919083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.919043    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:50.920732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.920700    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-kube-api-access-v7ghx" pod "e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" (UID: "e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8"). InnerVolumeSpecName "kube-api-access-v7ghx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:50.920839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:50.920710    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-xks-ca" pod "e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" (UID: "e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:51.020679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:51.020573    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:51.020679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:51.020616    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7ghx\" (UniqueName: \"kubernetes.io/projected/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8-kube-api-access-v7ghx\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:51.168157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:51.168117    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:51.736771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:51.736742    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"
Oct 30 22:27:51.736950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:51.736740    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q" event={"ID":"e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8","Type":"ContainerDied","Data":"d5a70d1b0e258f51aece9330f918c54cd33985a6da69550ce0669c60de3e6afb"}
Oct 30 22:27:51.736950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:51.736911    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5a70d1b0e258f51aece9330f918c54cd33985a6da69550ce0669c60de3e6afb"
Oct 30 22:27:55.539795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:55.539762    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:27:55.671825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:55.671797    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:55.916188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:55.916153    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:56.137813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:56.137780    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:57.591926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:57.591880    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"]
Oct 30 22:27:57.703553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:57.703516    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"]
Oct 30 22:27:57.971600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:57.971568    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:58.128157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:27:58.128121    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:27:58.141191 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:27:58.141130    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:28:00.458013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:00.457981    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:02.592509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.592453    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs" podUID="f508d76d-143e-4498-af63-9eee39e7336c" containerName="payload-processing" containerID="cri-o://454c273d83431728e319475d01c228d52848e646d941dfea17cb65dd0102e183" gracePeriod=25
Oct 30 22:28:02.704546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.704488    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp" podUID="5270aacc-de51-4e7c-9be2-f468c64cbf23" containerName="payload-pre-processing" containerID="cri-o://fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630" gracePeriod=25
Oct 30 22:28:02.817744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.817714    5503 generic.go:356] "Generic (PLEG): container finished" podID="f508d76d-143e-4498-af63-9eee39e7336c" containerID="454c273d83431728e319475d01c228d52848e646d941dfea17cb65dd0102e183" exitCode=0
Oct 30 22:28:02.817910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.817804    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs" event={"ID":"f508d76d-143e-4498-af63-9eee39e7336c","Type":"ContainerDied","Data":"454c273d83431728e319475d01c228d52848e646d941dfea17cb65dd0102e183"}
Oct 30 22:28:02.872503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.872478    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:28:02.974702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.974677    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f508d76d-143e-4498-af63-9eee39e7336c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f508d76d-143e-4498-af63-9eee39e7336c-plugins-config-volume\") pod \"f508d76d-143e-4498-af63-9eee39e7336c\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") "
Oct 30 22:28:02.974827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.974726    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f508d76d-143e-4498-af63-9eee39e7336c-kube-api-access-htxhf\" (UniqueName: \"kubernetes.io/projected/f508d76d-143e-4498-af63-9eee39e7336c-kube-api-access-htxhf\") pod \"f508d76d-143e-4498-af63-9eee39e7336c\" (UID: \"f508d76d-143e-4498-af63-9eee39e7336c\") "
Oct 30 22:28:02.975135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.975108    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f508d76d-143e-4498-af63-9eee39e7336c-plugins-config-volume" pod "f508d76d-143e-4498-af63-9eee39e7336c" (UID: "f508d76d-143e-4498-af63-9eee39e7336c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:02.976940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.976920    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f508d76d-143e-4498-af63-9eee39e7336c-kube-api-access-htxhf" pod "f508d76d-143e-4498-af63-9eee39e7336c" (UID: "f508d76d-143e-4498-af63-9eee39e7336c"). InnerVolumeSpecName "kube-api-access-htxhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:02.985253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:02.985234    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:28:03.075982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.075949    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5270aacc-de51-4e7c-9be2-f468c64cbf23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5270aacc-de51-4e7c-9be2-f468c64cbf23-plugins-config-volume\") pod \"5270aacc-de51-4e7c-9be2-f468c64cbf23\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") "
Oct 30 22:28:03.076167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.076079    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5270aacc-de51-4e7c-9be2-f468c64cbf23-kube-api-access-2sb2g\" (UniqueName: \"kubernetes.io/projected/5270aacc-de51-4e7c-9be2-f468c64cbf23-kube-api-access-2sb2g\") pod \"5270aacc-de51-4e7c-9be2-f468c64cbf23\" (UID: \"5270aacc-de51-4e7c-9be2-f468c64cbf23\") "
Oct 30 22:28:03.076340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.076315    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5270aacc-de51-4e7c-9be2-f468c64cbf23-plugins-config-volume" pod "5270aacc-de51-4e7c-9be2-f468c64cbf23" (UID: "5270aacc-de51-4e7c-9be2-f468c64cbf23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:03.076387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.076329    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f508d76d-143e-4498-af63-9eee39e7336c-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.076387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.076373    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxhf\" (UniqueName: \"kubernetes.io/projected/f508d76d-143e-4498-af63-9eee39e7336c-kube-api-access-htxhf\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.078198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.078178    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5270aacc-de51-4e7c-9be2-f468c64cbf23-kube-api-access-2sb2g" pod "5270aacc-de51-4e7c-9be2-f468c64cbf23" (UID: "5270aacc-de51-4e7c-9be2-f468c64cbf23"). InnerVolumeSpecName "kube-api-access-2sb2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:03.177421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.177377    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sb2g\" (UniqueName: \"kubernetes.io/projected/5270aacc-de51-4e7c-9be2-f468c64cbf23-kube-api-access-2sb2g\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.177421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.177416    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5270aacc-de51-4e7c-9be2-f468c64cbf23-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.380440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380403    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"]
Oct 30 22:28:03.380873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380859    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" containerName="revoke-keys"
Oct 30 22:28:03.380934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380877    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" containerName="revoke-keys"
Oct 30 22:28:03.380934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380899    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5270aacc-de51-4e7c-9be2-f468c64cbf23" containerName="payload-pre-processing"
Oct 30 22:28:03.380934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380906    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5270aacc-de51-4e7c-9be2-f468c64cbf23" containerName="payload-pre-processing"
Oct 30 22:28:03.380934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380927    5503 cpu_manager.go:414] "containerMap: removing container" podUID="f508d76d-143e-4498-af63-9eee39e7336c" containerName="payload-processing"
Oct 30 22:28:03.380934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380932    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f508d76d-143e-4498-af63-9eee39e7336c" containerName="payload-processing"
Oct 30 22:28:03.381087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380942    5503 cpu_manager.go:414] "containerMap: removing container" podUID="762d2b70-cf93-4379-a99a-906167aba75b" containerName="revoke-keys"
Oct 30 22:28:03.381087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.380949    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762d2b70-cf93-4379-a99a-906167aba75b" containerName="revoke-keys"
Oct 30 22:28:03.381087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.381023    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f508d76d-143e-4498-af63-9eee39e7336c" containerName="payload-processing"
Oct 30 22:28:03.381087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.381035    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="762d2b70-cf93-4379-a99a-906167aba75b" containerName="revoke-keys"
Oct 30 22:28:03.381087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.381045    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" containerName="revoke-keys"
Oct 30 22:28:03.381087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.381055    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5270aacc-de51-4e7c-9be2-f468c64cbf23" containerName="payload-pre-processing"
Oct 30 22:28:03.384339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.384323    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.438716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.438626    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:03.438889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.438830    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:28:03.439441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.439414    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:03.439505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.439456    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:03.439761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.439743    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"]
Oct 30 22:28:03.480390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.480360    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.480552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.480439    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.480552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.480493    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw65m\" (UniqueName: \"kubernetes.io/projected/38420e1d-f42d-43a1-8161-d6c08c501b14-kube-api-access-jw65m\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.581584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.581549    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.581780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.581604    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw65m\" (UniqueName: \"kubernetes.io/projected/38420e1d-f42d-43a1-8161-d6c08c501b14-kube-api-access-jw65m\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.581780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.581729    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.582334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.582303    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.583915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.583894    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.644117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.644082    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw65m\" (UniqueName: \"kubernetes.io/projected/38420e1d-f42d-43a1-8161-d6c08c501b14-kube-api-access-jw65m\") pod \"maas-api-revoke-sub-a947af20-zbn4b\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.697007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.696942    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:03.825074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.825052    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"
Oct 30 22:28:03.825224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.825047    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs" event={"ID":"f508d76d-143e-4498-af63-9eee39e7336c","Type":"ContainerDied","Data":"3513a08abcfb82ed16a6aaf80a194e6e8ea9287a8d5cd0b03605b600d2d789a4"}
Oct 30 22:28:03.825224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.825186    5503 scope.go:122] "RemoveContainer" containerID="454c273d83431728e319475d01c228d52848e646d941dfea17cb65dd0102e183"
Oct 30 22:28:03.826760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.826621    5503 generic.go:356] "Generic (PLEG): container finished" podID="5270aacc-de51-4e7c-9be2-f468c64cbf23" containerID="fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630" exitCode=0
Oct 30 22:28:03.826760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.826677    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp" event={"ID":"5270aacc-de51-4e7c-9be2-f468c64cbf23","Type":"ContainerDied","Data":"fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630"}
Oct 30 22:28:03.826760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.826722    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp" event={"ID":"5270aacc-de51-4e7c-9be2-f468c64cbf23","Type":"ContainerDied","Data":"c848fe9c3863481c0bacf01c7ba15bdbb89aeeac192719dd4cad58ccf841add4"}
Oct 30 22:28:03.826760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.826687    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"
Oct 30 22:28:03.835176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.835157    5503 scope.go:122] "RemoveContainer" containerID="fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630"
Oct 30 22:28:03.844119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.844100    5503 scope.go:122] "RemoveContainer" containerID="fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630"
Oct 30 22:28:03.844472 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:28:03.844449    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630\": container with ID starting with fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630 not found: ID does not exist" containerID="fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630"
Oct 30 22:28:03.844530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.844480    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630"} err="failed to get container status \"fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630\": rpc error: code = NotFound desc = could not find container \"fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630\": container with ID starting with fd743d578a3cf4c72e090e8a7775411aa28e70a5feb978bad8c5260c9987f630 not found: ID does not exist"
Oct 30 22:28:03.849499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.849478    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"]
Oct 30 22:28:03.854439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.854421    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:03.909611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.909577    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"]
Oct 30 22:28:03.986249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.986178    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:03.986249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.986227    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:03.986448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:03.986318    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwnb\" (UniqueName: \"kubernetes.io/projected/a29cc565-15a9-468c-b564-72ff57db8089-kube-api-access-srwnb\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.006997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.006971    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"]
Oct 30 22:28:04.020346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.020321    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ae8879-6b5d57574c-qbwjp"]
Oct 30 22:28:04.087486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.087456    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.087694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.087508    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.087694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.087535    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwnb\" (UniqueName: \"kubernetes.io/projected/a29cc565-15a9-468c-b564-72ff57db8089-kube-api-access-srwnb\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.088155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.088132    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.089763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.089736    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.110273 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:04.110248    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38420e1d_f42d_43a1_8161_d6c08c501b14.slice/crio-68ad80e82c09fcb19391b9711980755356a477e63a3488cef27e5baa06893c88 WatchSource:0}: Error finding container 68ad80e82c09fcb19391b9711980755356a477e63a3488cef27e5baa06893c88: Status 404 returned error can't find the container with id 68ad80e82c09fcb19391b9711980755356a477e63a3488cef27e5baa06893c88
Oct 30 22:28:04.113979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.113950    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"]
Oct 30 22:28:04.149482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.149455    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwnb\" (UniqueName: \"kubernetes.io/projected/a29cc565-15a9-468c-b564-72ff57db8089-kube-api-access-srwnb\") pod \"maas-api-revoke-sub-9331bb73-xfhq4\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.169320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.169289    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:04.270795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.270708    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"]
Oct 30 22:28:04.344582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.344552    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ae8879-594df77f75-qv5xs"]
Oct 30 22:28:04.545591 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:04.545561    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29cc565_15a9_468c_b564_72ff57db8089.slice/crio-65d0f52ddfd2a325949cc42733cadd86ac3a000097a3ebad391184a070cfaed7 WatchSource:0}: Error finding container 65d0f52ddfd2a325949cc42733cadd86ac3a000097a3ebad391184a070cfaed7: Status 404 returned error can't find the container with id 65d0f52ddfd2a325949cc42733cadd86ac3a000097a3ebad391184a070cfaed7
Oct 30 22:28:04.547342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.547309    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"]
Oct 30 22:28:04.834675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.834615    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b" event={"ID":"38420e1d-f42d-43a1-8161-d6c08c501b14","Type":"ContainerStarted","Data":"e3a82f8e39a77e75f7bac45b92a38b30f4502dc3d1f8f834ff149e0fa7642206"}
Oct 30 22:28:04.835381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.834705    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b" event={"ID":"38420e1d-f42d-43a1-8161-d6c08c501b14","Type":"ContainerStarted","Data":"68ad80e82c09fcb19391b9711980755356a477e63a3488cef27e5baa06893c88"}
Oct 30 22:28:04.836242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.836214    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4" event={"ID":"a29cc565-15a9-468c-b564-72ff57db8089","Type":"ContainerStarted","Data":"0670c0e9abbe38b1ae4c9a1e64cb70715593623ff794b36c2685be5ad4fc4746"}
Oct 30 22:28:04.836356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.836262    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4" event={"ID":"a29cc565-15a9-468c-b564-72ff57db8089","Type":"ContainerStarted","Data":"65d0f52ddfd2a325949cc42733cadd86ac3a000097a3ebad391184a070cfaed7"}
Oct 30 22:28:04.954005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.953954    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b" podStartSLOduration=1.953939807 podStartE2EDuration="1.953939807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:04.952842263 +0000 UTC m=+2461.287419031" watchObservedRunningTime="2026-10-30 22:28:04.953939807 +0000 UTC m=+2461.288516576"
Oct 30 22:28:04.966793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:04.966771    5503 scope.go:122] "RemoveContainer" containerID="5ccc6c046af78faeefe07d0f499d25b757dffc30999d2ca7c3fa796b0b00c60e"
Oct 30 22:28:05.100489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.100396    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4" podStartSLOduration=2.100383545 podStartE2EDuration="2.100383545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:05.099989683 +0000 UTC m=+2461.434566453" watchObservedRunningTime="2026-10-30 22:28:05.100383545 +0000 UTC m=+2461.434960314"
Oct 30 22:28:05.147608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.147579    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:05.789009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.788933    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5270aacc-de51-4e7c-9be2-f468c64cbf23" path="/var/lib/kubelet/pods/5270aacc-de51-4e7c-9be2-f468c64cbf23/volumes"
Oct 30 22:28:05.789374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.789360    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f508d76d-143e-4498-af63-9eee39e7336c" path="/var/lib/kubelet/pods/f508d76d-143e-4498-af63-9eee39e7336c/volumes"
Oct 30 22:28:05.845575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.845540    5503 generic.go:356] "Generic (PLEG): container finished" podID="a29cc565-15a9-468c-b564-72ff57db8089" containerID="0670c0e9abbe38b1ae4c9a1e64cb70715593623ff794b36c2685be5ad4fc4746" exitCode=0
Oct 30 22:28:05.845960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.845622    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4" event={"ID":"a29cc565-15a9-468c-b564-72ff57db8089","Type":"ContainerDied","Data":"0670c0e9abbe38b1ae4c9a1e64cb70715593623ff794b36c2685be5ad4fc4746"}
Oct 30 22:28:05.847333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.847302    5503 generic.go:356] "Generic (PLEG): container finished" podID="38420e1d-f42d-43a1-8161-d6c08c501b14" containerID="e3a82f8e39a77e75f7bac45b92a38b30f4502dc3d1f8f834ff149e0fa7642206" exitCode=0
Oct 30 22:28:05.847461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:05.847343    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b" event={"ID":"38420e1d-f42d-43a1-8161-d6c08c501b14","Type":"ContainerDied","Data":"e3a82f8e39a77e75f7bac45b92a38b30f4502dc3d1f8f834ff149e0fa7642206"}
Oct 30 22:28:07.025908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.025880    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:07.030198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.030180    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:07.118909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.118808    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-xks-ca\") pod \"38420e1d-f42d-43a1-8161-d6c08c501b14\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") "
Oct 30 22:28:07.119090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.118915    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-openshift-ca\") pod \"38420e1d-f42d-43a1-8161-d6c08c501b14\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") "
Oct 30 22:28:07.119090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.118963    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-openshift-ca\") pod \"a29cc565-15a9-468c-b564-72ff57db8089\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") "
Oct 30 22:28:07.119090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.119059    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29cc565-15a9-468c-b564-72ff57db8089-kube-api-access-srwnb\" (UniqueName: \"kubernetes.io/projected/a29cc565-15a9-468c-b564-72ff57db8089-kube-api-access-srwnb\") pod \"a29cc565-15a9-468c-b564-72ff57db8089\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") "
Oct 30 22:28:07.119248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.119099    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-xks-ca\") pod \"a29cc565-15a9-468c-b564-72ff57db8089\" (UID: \"a29cc565-15a9-468c-b564-72ff57db8089\") "
Oct 30 22:28:07.119248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.119197    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38420e1d-f42d-43a1-8161-d6c08c501b14-kube-api-access-jw65m\" (UniqueName: \"kubernetes.io/projected/38420e1d-f42d-43a1-8161-d6c08c501b14-kube-api-access-jw65m\") pod \"38420e1d-f42d-43a1-8161-d6c08c501b14\" (UID: \"38420e1d-f42d-43a1-8161-d6c08c501b14\") "
Oct 30 22:28:07.119346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.119310    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-openshift-ca" pod "38420e1d-f42d-43a1-8161-d6c08c501b14" (UID: "38420e1d-f42d-43a1-8161-d6c08c501b14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:07.119395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.119343    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-openshift-ca" pod "a29cc565-15a9-468c-b564-72ff57db8089" (UID: "a29cc565-15a9-468c-b564-72ff57db8089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:07.119654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.119612    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:07.119799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.119661    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:07.121047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.121020    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-xks-ca" pod "38420e1d-f42d-43a1-8161-d6c08c501b14" (UID: "38420e1d-f42d-43a1-8161-d6c08c501b14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:07.121192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.121174    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-xks-ca" pod "a29cc565-15a9-468c-b564-72ff57db8089" (UID: "a29cc565-15a9-468c-b564-72ff57db8089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:07.121571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.121549    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38420e1d-f42d-43a1-8161-d6c08c501b14-kube-api-access-jw65m" pod "38420e1d-f42d-43a1-8161-d6c08c501b14" (UID: "38420e1d-f42d-43a1-8161-d6c08c501b14"). InnerVolumeSpecName "kube-api-access-jw65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:07.121675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.121584    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29cc565-15a9-468c-b564-72ff57db8089-kube-api-access-srwnb" pod "a29cc565-15a9-468c-b564-72ff57db8089" (UID: "a29cc565-15a9-468c-b564-72ff57db8089"). InnerVolumeSpecName "kube-api-access-srwnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:07.220653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.220606    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cc565-15a9-468c-b564-72ff57db8089-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:07.220653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.220661    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw65m\" (UniqueName: \"kubernetes.io/projected/38420e1d-f42d-43a1-8161-d6c08c501b14-kube-api-access-jw65m\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:07.220864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.220672    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38420e1d-f42d-43a1-8161-d6c08c501b14-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:07.220864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.220682    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srwnb\" (UniqueName: \"kubernetes.io/projected/a29cc565-15a9-468c-b564-72ff57db8089-kube-api-access-srwnb\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:07.869300 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.869261    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b" event={"ID":"38420e1d-f42d-43a1-8161-d6c08c501b14","Type":"ContainerDied","Data":"68ad80e82c09fcb19391b9711980755356a477e63a3488cef27e5baa06893c88"}
Oct 30 22:28:07.869457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.869354    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ad80e82c09fcb19391b9711980755356a477e63a3488cef27e5baa06893c88"
Oct 30 22:28:07.869457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.869427    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"
Oct 30 22:28:07.874329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.874294    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4" event={"ID":"a29cc565-15a9-468c-b564-72ff57db8089","Type":"ContainerDied","Data":"65d0f52ddfd2a325949cc42733cadd86ac3a000097a3ebad391184a070cfaed7"}
Oct 30 22:28:07.874519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.874359    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65d0f52ddfd2a325949cc42733cadd86ac3a000097a3ebad391184a070cfaed7"
Oct 30 22:28:07.874519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:07.874306    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"
Oct 30 22:28:09.097434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:09.097401    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.548657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:10.548598    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.702453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:10.702417    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:10.861911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:10.861818    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.016802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:11.016763    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"]
Oct 30 22:28:11.020746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:11.020715    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.067453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:11.067426    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ae8879-7k52z"]
Oct 30 22:28:11.447277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:11.447238    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:11.789607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:11.789527    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762d2b70-cf93-4379-a99a-906167aba75b" path="/var/lib/kubelet/pods/762d2b70-cf93-4379-a99a-906167aba75b/volumes"
Oct 30 22:28:14.708360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.708322    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"]
Oct 30 22:28:14.709032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.709010    5503 cpu_manager.go:414] "containerMap: removing container" podUID="38420e1d-f42d-43a1-8161-d6c08c501b14" containerName="revoke-keys"
Oct 30 22:28:14.709137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.709033    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38420e1d-f42d-43a1-8161-d6c08c501b14" containerName="revoke-keys"
Oct 30 22:28:14.709137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.709047    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a29cc565-15a9-468c-b564-72ff57db8089" containerName="revoke-keys"
Oct 30 22:28:14.709137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.709056    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29cc565-15a9-468c-b564-72ff57db8089" containerName="revoke-keys"
Oct 30 22:28:14.709288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.709169    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="38420e1d-f42d-43a1-8161-d6c08c501b14" containerName="revoke-keys"
Oct 30 22:28:14.709288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.709184    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29cc565-15a9-468c-b564-72ff57db8089" containerName="revoke-keys"
Oct 30 22:28:14.713779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.713744    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.767820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.767788    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"]
Oct 30 22:28:14.815801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.815757    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:14.815960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.815808    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:14.815960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.815833    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:14.815960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.815886    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:28:14.817515 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.817493    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.817618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.817525    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.817618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.817548    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvzc\" (UniqueName: \"kubernetes.io/projected/7d5b032b-dd33-43f1-89ab-71f4949bccbe-kube-api-access-frvzc\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.919048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.919010    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.919246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.919055    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.919246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.919092    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvzc\" (UniqueName: \"kubernetes.io/projected/7d5b032b-dd33-43f1-89ab-71f4949bccbe-kube-api-access-frvzc\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.919772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.919752    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.921459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.921437    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:14.982447 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:14.982375    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvzc\" (UniqueName: \"kubernetes.io/projected/7d5b032b-dd33-43f1-89ab-71f4949bccbe-kube-api-access-frvzc\") pod \"maas-api-revoke-sub-baf03ea8-wqff4\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:15.027016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:15.026980    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:15.399886 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:15.399856    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5b032b_dd33_43f1_89ab_71f4949bccbe.slice/crio-69865380a35c32a2cb9f6a457405bd75b2fe2e8a5d63b3662b41faaac284bbaa WatchSource:0}: Error finding container 69865380a35c32a2cb9f6a457405bd75b2fe2e8a5d63b3662b41faaac284bbaa: Status 404 returned error can't find the container with id 69865380a35c32a2cb9f6a457405bd75b2fe2e8a5d63b3662b41faaac284bbaa
Oct 30 22:28:15.402560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:15.402532    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"]
Oct 30 22:28:15.715265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:15.715231    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:15.948900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:15.948865    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4" event={"ID":"7d5b032b-dd33-43f1-89ab-71f4949bccbe","Type":"ContainerStarted","Data":"1c4cb89d0c4def6633897cd66413d6292ec5ceb949bb91d0383b017206ce92af"}
Oct 30 22:28:15.949063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:15.948932    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4" event={"ID":"7d5b032b-dd33-43f1-89ab-71f4949bccbe","Type":"ContainerStarted","Data":"69865380a35c32a2cb9f6a457405bd75b2fe2e8a5d63b3662b41faaac284bbaa"}
Oct 30 22:28:15.978015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:15.977954    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:16.070115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:16.070058    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4" podStartSLOduration=2.070043256 podStartE2EDuration="2.070043256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:16.068433109 +0000 UTC m=+2472.403009901" watchObservedRunningTime="2026-10-30 22:28:16.070043256 +0000 UTC m=+2472.404620025"
Oct 30 22:28:16.963020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:16.962988    5503 generic.go:356] "Generic (PLEG): container finished" podID="7d5b032b-dd33-43f1-89ab-71f4949bccbe" containerID="1c4cb89d0c4def6633897cd66413d6292ec5ceb949bb91d0383b017206ce92af" exitCode=0
Oct 30 22:28:16.963430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:16.963072    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4" event={"ID":"7d5b032b-dd33-43f1-89ab-71f4949bccbe","Type":"ContainerDied","Data":"1c4cb89d0c4def6633897cd66413d6292ec5ceb949bb91d0383b017206ce92af"}
Oct 30 22:28:18.107289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.107263    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:18.153270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.153241    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-xks-ca\") pod \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") "
Oct 30 22:28:18.153426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.153286    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5b032b-dd33-43f1-89ab-71f4949bccbe-kube-api-access-frvzc\" (UniqueName: \"kubernetes.io/projected/7d5b032b-dd33-43f1-89ab-71f4949bccbe-kube-api-access-frvzc\") pod \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") "
Oct 30 22:28:18.153426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.153410    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-openshift-ca\") pod \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\" (UID: \"7d5b032b-dd33-43f1-89ab-71f4949bccbe\") "
Oct 30 22:28:18.153768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.153738    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-openshift-ca" pod "7d5b032b-dd33-43f1-89ab-71f4949bccbe" (UID: "7d5b032b-dd33-43f1-89ab-71f4949bccbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:18.153972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.153952    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:18.155334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.155313    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-xks-ca" pod "7d5b032b-dd33-43f1-89ab-71f4949bccbe" (UID: "7d5b032b-dd33-43f1-89ab-71f4949bccbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:18.155452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.155437    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5b032b-dd33-43f1-89ab-71f4949bccbe-kube-api-access-frvzc" pod "7d5b032b-dd33-43f1-89ab-71f4949bccbe" (UID: "7d5b032b-dd33-43f1-89ab-71f4949bccbe"). InnerVolumeSpecName "kube-api-access-frvzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:18.255142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.255070    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5b032b-dd33-43f1-89ab-71f4949bccbe-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:18.255142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.255099    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvzc\" (UniqueName: \"kubernetes.io/projected/7d5b032b-dd33-43f1-89ab-71f4949bccbe-kube-api-access-frvzc\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:18.319097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.319042    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.488797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.488763    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.662006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.661964    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"]
Oct 30 22:28:18.662320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.662229    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy" containerID="cri-o://1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee" gracePeriod=30
Oct 30 22:28:18.673973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.673945    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"]
Oct 30 22:28:18.674202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.674163    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy" containerID="cri-o://09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573" gracePeriod=30
Oct 30 22:28:18.679248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.679208    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:18.976223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.976144    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4" event={"ID":"7d5b032b-dd33-43f1-89ab-71f4949bccbe","Type":"ContainerDied","Data":"69865380a35c32a2cb9f6a457405bd75b2fe2e8a5d63b3662b41faaac284bbaa"}
Oct 30 22:28:18.976223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.976180    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"
Oct 30 22:28:18.976223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:18.976192    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69865380a35c32a2cb9f6a457405bd75b2fe2e8a5d63b3662b41faaac284bbaa"
Oct 30 22:28:19.551045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.551013    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"]
Oct 30 22:28:19.551565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.551493    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7d5b032b-dd33-43f1-89ab-71f4949bccbe" containerName="revoke-keys"
Oct 30 22:28:19.551565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.551508    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5b032b-dd33-43f1-89ab-71f4949bccbe" containerName="revoke-keys"
Oct 30 22:28:19.551720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.551593    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5b032b-dd33-43f1-89ab-71f4949bccbe" containerName="revoke-keys"
Oct 30 22:28:19.555077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.555056    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.569203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.569174    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnwc5\" (UniqueName: \"kubernetes.io/projected/d931c45b-62ed-41b4-a59b-e39e62752027-kube-api-access-mnwc5\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.569352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.569330    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.569414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.569383    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.610834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.610803    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"]
Oct 30 22:28:19.611200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.611179    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:28:19.611297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.611232    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:19.611349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.611299    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:19.611404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.611348    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:19.670106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.670076    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.670280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.670145    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.670280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.670249    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnwc5\" (UniqueName: \"kubernetes.io/projected/d931c45b-62ed-41b4-a59b-e39e62752027-kube-api-access-mnwc5\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.670896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.670871    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.678060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.678033    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.737514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.737480    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnwc5\" (UniqueName: \"kubernetes.io/projected/d931c45b-62ed-41b4-a59b-e39e62752027-kube-api-access-mnwc5\") pod \"maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:19.869210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:19.869125    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:20.244231 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:20.244187    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd931c45b_62ed_41b4_a59b_e39e62752027.slice/crio-d7f1537683fd956f8c49246e073cb8e2a37bd1441088a5ca9354790c5d63ed22 WatchSource:0}: Error finding container d7f1537683fd956f8c49246e073cb8e2a37bd1441088a5ca9354790c5d63ed22: Status 404 returned error can't find the container with id d7f1537683fd956f8c49246e073cb8e2a37bd1441088a5ca9354790c5d63ed22
Oct 30 22:28:20.248115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:20.248091    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"]
Oct 30 22:28:20.487411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:20.487378    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:20.991524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:20.991489    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr" event={"ID":"d931c45b-62ed-41b4-a59b-e39e62752027","Type":"ContainerStarted","Data":"cc4cc92a2fcefae8b1a1687bcc496a1924e4af084dc3a623c4b0194a4c3cf928"}
Oct 30 22:28:20.992135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:20.991583    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr" event={"ID":"d931c45b-62ed-41b4-a59b-e39e62752027","Type":"ContainerStarted","Data":"d7f1537683fd956f8c49246e073cb8e2a37bd1441088a5ca9354790c5d63ed22"}
Oct 30 22:28:21.111380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:21.111326    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr" podStartSLOduration=2.111313773 podStartE2EDuration="2.111313773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:21.10929568 +0000 UTC m=+2477.443872448" watchObservedRunningTime="2026-10-30 22:28:21.111313773 +0000 UTC m=+2477.445890579"
Oct 30 22:28:21.998682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:21.998648    5503 generic.go:356] "Generic (PLEG): container finished" podID="d931c45b-62ed-41b4-a59b-e39e62752027" containerID="cc4cc92a2fcefae8b1a1687bcc496a1924e4af084dc3a623c4b0194a4c3cf928" exitCode=0
Oct 30 22:28:21.999064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:21.998682    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr" event={"ID":"d931c45b-62ed-41b4-a59b-e39e62752027","Type":"ContainerDied","Data":"cc4cc92a2fcefae8b1a1687bcc496a1924e4af084dc3a623c4b0194a4c3cf928"}
Oct 30 22:28:23.148554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.148526    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:23.212677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.212617    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-openshift-ca\") pod \"d931c45b-62ed-41b4-a59b-e39e62752027\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") "
Oct 30 22:28:23.212833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.212756    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d931c45b-62ed-41b4-a59b-e39e62752027-kube-api-access-mnwc5\" (UniqueName: \"kubernetes.io/projected/d931c45b-62ed-41b4-a59b-e39e62752027-kube-api-access-mnwc5\") pod \"d931c45b-62ed-41b4-a59b-e39e62752027\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") "
Oct 30 22:28:23.212833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.212819    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-xks-ca\") pod \"d931c45b-62ed-41b4-a59b-e39e62752027\" (UID: \"d931c45b-62ed-41b4-a59b-e39e62752027\") "
Oct 30 22:28:23.212974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.212946    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-openshift-ca" pod "d931c45b-62ed-41b4-a59b-e39e62752027" (UID: "d931c45b-62ed-41b4-a59b-e39e62752027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:23.213333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.213301    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:23.214933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.214911    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d931c45b-62ed-41b4-a59b-e39e62752027-kube-api-access-mnwc5" pod "d931c45b-62ed-41b4-a59b-e39e62752027" (UID: "d931c45b-62ed-41b4-a59b-e39e62752027"). InnerVolumeSpecName "kube-api-access-mnwc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:23.215054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.214979    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-xks-ca" pod "d931c45b-62ed-41b4-a59b-e39e62752027" (UID: "d931c45b-62ed-41b4-a59b-e39e62752027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:23.314361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.314290    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnwc5\" (UniqueName: \"kubernetes.io/projected/d931c45b-62ed-41b4-a59b-e39e62752027-kube-api-access-mnwc5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:23.314361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.314319    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d931c45b-62ed-41b4-a59b-e39e62752027-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:23.747091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.747054    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:23.924373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.924300    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:28:23.958681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:23.958652    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:28:24.012430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.012393    5503 generic.go:356] "Generic (PLEG): container finished" podID="2c297409-3d79-40ba-b6c3-d883887f7933" containerID="09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573" exitCode=0
Oct 30 22:28:24.012617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.012444    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"
Oct 30 22:28:24.012617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.012482    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" event={"ID":"2c297409-3d79-40ba-b6c3-d883887f7933","Type":"ContainerDied","Data":"09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573"}
Oct 30 22:28:24.012617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.012545    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv" event={"ID":"2c297409-3d79-40ba-b6c3-d883887f7933","Type":"ContainerDied","Data":"e1ce1d8fbb6c2814bc7b374ab56281a584d94df84c053d7544017a11f6a7b5b5"}
Oct 30 22:28:24.012617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.012574    5503 scope.go:122] "RemoveContainer" containerID="09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573"
Oct 30 22:28:24.013985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.013967    5503 generic.go:356] "Generic (PLEG): container finished" podID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerID="1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee" exitCode=0
Oct 30 22:28:24.014048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.013992    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" event={"ID":"11341b43-f930-4bdd-8a93-0c4ed09e484c","Type":"ContainerDied","Data":"1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee"}
Oct 30 22:28:24.014048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.014016    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"
Oct 30 22:28:24.014048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.014031    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs" event={"ID":"11341b43-f930-4bdd-8a93-0c4ed09e484c","Type":"ContainerDied","Data":"2dfca6d29b320ab352e50ed66de84d681f241977363ebfe34726324ff6d7a0d7"}
Oct 30 22:28:24.015468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.015452    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"
Oct 30 22:28:24.015468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.015463    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr" event={"ID":"d931c45b-62ed-41b4-a59b-e39e62752027","Type":"ContainerDied","Data":"d7f1537683fd956f8c49246e073cb8e2a37bd1441088a5ca9354790c5d63ed22"}
Oct 30 22:28:24.015606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.015492    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f1537683fd956f8c49246e073cb8e2a37bd1441088a5ca9354790c5d63ed22"
Oct 30 22:28:24.019506 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019485    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-istio-token\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-istio-token\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.019607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019530    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c297409-3d79-40ba-b6c3-d883887f7933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c297409-3d79-40ba-b6c3-d883887f7933-istio-podinfo\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.019607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019554    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-kube-api-access-q6rh6\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-kube-api-access-q6rh6\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.019607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019589    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-kube-api-access-47kzl\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-kube-api-access-47kzl\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.019801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019611    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-credential-socket\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.019801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019660    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-socket\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.019801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019694    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-socket\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.019801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019738    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-envoy\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.019801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019768    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-podinfo\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019803    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-data\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019844    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c297409-3d79-40ba-b6c3-d883887f7933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c297409-3d79-40ba-b6c3-d883887f7933-istiod-ca-cert\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019873    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-certs\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019903    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-token\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-token\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019937    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-envoy\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019969    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-socket" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019987    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-socket" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.019974    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11341b43-f930-4bdd-8a93-0c4ed09e484c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11341b43-f930-4bdd-8a93-0c4ed09e484c-istiod-ca-cert\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.020085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020065    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-data\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.020558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020180    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-certs\") pod \"2c297409-3d79-40ba-b6c3-d883887f7933\" (UID: \"2c297409-3d79-40ba-b6c3-d883887f7933\") "
Oct 30 22:28:24.020558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020192    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-credential-socket" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.020558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020239    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-credential-socket\") pod \"11341b43-f930-4bdd-8a93-0c4ed09e484c\" (UID: \"11341b43-f930-4bdd-8a93-0c4ed09e484c\") "
Oct 30 22:28:24.020558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020298    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11341b43-f930-4bdd-8a93-0c4ed09e484c-istiod-ca-cert" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:24.020558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020376    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-data" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.020871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020581    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-certs" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.020871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020804    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11341b43-f930-4bdd-8a93-0c4ed09e484c-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.020871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020825    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.020871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020840    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.020871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020857    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.020871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020872    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.021241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020887    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.021241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.020942    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-data" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.021241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.021003    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-certs" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.021241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.021059    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c297409-3d79-40ba-b6c3-d883887f7933-istiod-ca-cert" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:24.021480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.021295    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-credential-socket" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.023061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.023025    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-istio-token" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:24.023218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.023175    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-podinfo" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:24.023308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.023240    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-kube-api-access-q6rh6" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "kube-api-access-q6rh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:24.023308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.023260    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-token" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:24.023391 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.023334    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-envoy" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.023513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.023496    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-envoy" pod "11341b43-f930-4bdd-8a93-0c4ed09e484c" (UID: "11341b43-f930-4bdd-8a93-0c4ed09e484c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:24.024208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.024186    5503 scope.go:122] "RemoveContainer" containerID="09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573"
Oct 30 22:28:24.024266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.024254    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-kube-api-access-47kzl" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "kube-api-access-47kzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:24.024492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.024471    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c297409-3d79-40ba-b6c3-d883887f7933-istio-podinfo" pod "2c297409-3d79-40ba-b6c3-d883887f7933" (UID: "2c297409-3d79-40ba-b6c3-d883887f7933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:24.024492 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:28:24.024487    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573\": container with ID starting with 09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573 not found: ID does not exist" containerID="09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573"
Oct 30 22:28:24.024662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.024509    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573"} err="failed to get container status \"09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573\": rpc error: code = NotFound desc = could not find container \"09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573\": container with ID starting with 09a7cb01d0a67bb71f953c13f1269ed26d6fb1f97484be728879764e28650573 not found: ID does not exist"
Oct 30 22:28:24.024662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.024528    5503 scope.go:122] "RemoveContainer" containerID="1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee"
Oct 30 22:28:24.033527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.033510    5503 scope.go:122] "RemoveContainer" containerID="1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee"
Oct 30 22:28:24.033828 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:28:24.033811    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee\": container with ID starting with 1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee not found: ID does not exist" containerID="1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee"
Oct 30 22:28:24.033884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.033839    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee"} err="failed to get container status \"1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee\": rpc error: code = NotFound desc = could not find container \"1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee\": container with ID starting with 1d785a7c03767c11a865ec1f4a00e681825362004e8322b59bf451bb10e407ee not found: ID does not exist"
Oct 30 22:28:24.121650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121603    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121649    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121659    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121669    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c297409-3d79-40ba-b6c3-d883887f7933-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121680    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6rh6\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-kube-api-access-q6rh6\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121690    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47kzl\" (UniqueName: \"kubernetes.io/projected/2c297409-3d79-40ba-b6c3-d883887f7933-kube-api-access-47kzl\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121700    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121711    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121720    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121728    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c297409-3d79-40ba-b6c3-d883887f7933-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121736    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11341b43-f930-4bdd-8a93-0c4ed09e484c-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.121843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.121745    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c297409-3d79-40ba-b6c3-d883887f7933-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:24.445112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.445078    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"]
Oct 30 22:28:24.453354 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.453328    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-4tldv"]
Oct 30 22:28:24.565302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.565274    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"]
Oct 30 22:28:24.569429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.569407    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ae8879-openshift-default-5bc8464785-bg5qs"]
Oct 30 22:28:24.892798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:24.892764    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:25.540815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.540780    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"]
Oct 30 22:28:25.541223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541205    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy"
Oct 30 22:28:25.541223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541216    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy"
Oct 30 22:28:25.541323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541231    5503 cpu_manager.go:414] "containerMap: removing container" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy"
Oct 30 22:28:25.541323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541236    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy"
Oct 30 22:28:25.541323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541254    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d931c45b-62ed-41b4-a59b-e39e62752027" containerName="revoke-keys"
Oct 30 22:28:25.541323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541260    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d931c45b-62ed-41b4-a59b-e39e62752027" containerName="revoke-keys"
Oct 30 22:28:25.541450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541330    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" containerName="istio-proxy"
Oct 30 22:28:25.541450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541341    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d931c45b-62ed-41b4-a59b-e39e62752027" containerName="revoke-keys"
Oct 30 22:28:25.541450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.541353    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" containerName="istio-proxy"
Oct 30 22:28:25.545981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.545965    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.592334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.592274    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:25.598438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.598411    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"]
Oct 30 22:28:25.599418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.599392    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:25.645556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.645193    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.645556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.645250    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.645556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.645425    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgwp\" (UniqueName: \"kubernetes.io/projected/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-kube-api-access-ktgwp\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.646782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.646756    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:28:25.646934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.646852    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:25.646934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.646871    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:25.746502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.746469    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgwp\" (UniqueName: \"kubernetes.io/projected/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-kube-api-access-ktgwp\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.746700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.746629    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.746747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.746708    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.747312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.747293    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.748871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.748852    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.788563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.788531    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11341b43-f930-4bdd-8a93-0c4ed09e484c" path="/var/lib/kubelet/pods/11341b43-f930-4bdd-8a93-0c4ed09e484c/volumes"
Oct 30 22:28:25.789138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.789111    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c297409-3d79-40ba-b6c3-d883887f7933" path="/var/lib/kubelet/pods/2c297409-3d79-40ba-b6c3-d883887f7933/volumes"
Oct 30 22:28:25.809123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.809062    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgwp\" (UniqueName: \"kubernetes.io/projected/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-kube-api-access-ktgwp\") pod \"maas-api-revoke-sub-b4d2f6b1-l88zn\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:25.862018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:25.861987    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:26.234535 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:26.234494    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a18050e_6f0e_4504_8fc3_6cf63e0f89f9.slice/crio-7b05f073c9c8bf9cd9277d08d6c6db3db615ed10230b54c4596deb977e1bfae4 WatchSource:0}: Error finding container 7b05f073c9c8bf9cd9277d08d6c6db3db615ed10230b54c4596deb977e1bfae4: Status 404 returned error can't find the container with id 7b05f073c9c8bf9cd9277d08d6c6db3db615ed10230b54c4596deb977e1bfae4
Oct 30 22:28:26.238576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:26.238551    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"]
Oct 30 22:28:27.037083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:27.037051    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn" event={"ID":"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9","Type":"ContainerStarted","Data":"0320abbd4e2d339a29dd9fd6ec1a4f41f064263409916d33738cbc19fc158403"}
Oct 30 22:28:27.037477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:27.037104    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn" event={"ID":"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9","Type":"ContainerStarted","Data":"7b05f073c9c8bf9cd9277d08d6c6db3db615ed10230b54c4596deb977e1bfae4"}
Oct 30 22:28:27.155498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:27.155444    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn" podStartSLOduration=2.155429309 podStartE2EDuration="2.155429309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:27.15423398 +0000 UTC m=+2483.488810750" watchObservedRunningTime="2026-10-30 22:28:27.155429309 +0000 UTC m=+2483.490006078"
Oct 30 22:28:28.045088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:28.045054    5503 generic.go:356] "Generic (PLEG): container finished" podID="5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" containerID="0320abbd4e2d339a29dd9fd6ec1a4f41f064263409916d33738cbc19fc158403" exitCode=0
Oct 30 22:28:28.045487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:28.045096    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn" event={"ID":"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9","Type":"ContainerDied","Data":"0320abbd4e2d339a29dd9fd6ec1a4f41f064263409916d33738cbc19fc158403"}
Oct 30 22:28:28.098869 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:28:28.098827    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:28:29.204359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.204330    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:29.284903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.284868    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-openshift-ca\") pod \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") "
Oct 30 22:28:29.285105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.284953    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-kube-api-access-ktgwp\" (UniqueName: \"kubernetes.io/projected/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-kube-api-access-ktgwp\") pod \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") "
Oct 30 22:28:29.285105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.284974    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-xks-ca\") pod \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\" (UID: \"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9\") "
Oct 30 22:28:29.285301 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.285279    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-openshift-ca" pod "5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" (UID: "5a18050e-6f0e-4504-8fc3-6cf63e0f89f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:29.287054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.287031    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-xks-ca" pod "5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" (UID: "5a18050e-6f0e-4504-8fc3-6cf63e0f89f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:29.287054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.287041    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-kube-api-access-ktgwp" pod "5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" (UID: "5a18050e-6f0e-4504-8fc3-6cf63e0f89f9"). InnerVolumeSpecName "kube-api-access-ktgwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:29.386695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.386594    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktgwp\" (UniqueName: \"kubernetes.io/projected/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-kube-api-access-ktgwp\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.386695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.386622    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:29.386695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:29.386632    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:30.059576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:30.059497    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"
Oct 30 22:28:30.059576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:30.059525    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn" event={"ID":"5a18050e-6f0e-4504-8fc3-6cf63e0f89f9","Type":"ContainerDied","Data":"7b05f073c9c8bf9cd9277d08d6c6db3db615ed10230b54c4596deb977e1bfae4"}
Oct 30 22:28:30.059576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:30.059571    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b05f073c9c8bf9cd9277d08d6c6db3db615ed10230b54c4596deb977e1bfae4"
Oct 30 22:28:30.500659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:30.500607    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:31.910044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:31.910011    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:32.060012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:32.059975    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:35.200971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:35.200929    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:36.595164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.595128    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"]
Oct 30 22:28:36.595583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.595569    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" containerName="revoke-keys"
Oct 30 22:28:36.595665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.595585    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" containerName="revoke-keys"
Oct 30 22:28:36.595727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.595716    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" containerName="revoke-keys"
Oct 30 22:28:36.599217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.599199    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.655351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.655311    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"]
Oct 30 22:28:36.669250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.669218    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de45cc76-477a-4813-af22-3007a3748e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.669464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.669443    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de45cc76-477a-4813-af22-3007a3748e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.669574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.669551    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jxh\" (UniqueName: \"kubernetes.io/projected/de45cc76-477a-4813-af22-3007a3748e10-kube-api-access-k9jxh\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.702393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.702363    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:36.702393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.702363    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:36.702393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.702366    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:36.703163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.703146    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:28:36.770791 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.770764    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de45cc76-477a-4813-af22-3007a3748e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.770950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.770886    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de45cc76-477a-4813-af22-3007a3748e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.770950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.770932    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jxh\" (UniqueName: \"kubernetes.io/projected/de45cc76-477a-4813-af22-3007a3748e10-kube-api-access-k9jxh\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.771590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.771565    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de45cc76-477a-4813-af22-3007a3748e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.773259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.773234    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de45cc76-477a-4813-af22-3007a3748e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.832967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.832945    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jxh\" (UniqueName: \"kubernetes.io/projected/de45cc76-477a-4813-af22-3007a3748e10-kube-api-access-k9jxh\") pod \"maas-api-revoke-sub-84d74c19-2wvrq\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:36.912148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:36.912117    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:37.291581 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:37.291548    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde45cc76_477a_4813_af22_3007a3748e10.slice/crio-88549266f44cdcca85fd80c90112dfcd5735d61cc73bfbd2dab9e2e62460eb56 WatchSource:0}: Error finding container 88549266f44cdcca85fd80c90112dfcd5735d61cc73bfbd2dab9e2e62460eb56: Status 404 returned error can't find the container with id 88549266f44cdcca85fd80c90112dfcd5735d61cc73bfbd2dab9e2e62460eb56
Oct 30 22:28:37.293792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:37.293766    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"]
Oct 30 22:28:38.111009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:38.110970    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq" event={"ID":"de45cc76-477a-4813-af22-3007a3748e10","Type":"ContainerStarted","Data":"d1855e6e670b933a88524ad28ae7f1b2e4d14cbe4345b4e464e0e05254396a4d"}
Oct 30 22:28:38.111380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:38.111033    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq" event={"ID":"de45cc76-477a-4813-af22-3007a3748e10","Type":"ContainerStarted","Data":"88549266f44cdcca85fd80c90112dfcd5735d61cc73bfbd2dab9e2e62460eb56"}
Oct 30 22:28:38.229768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:38.229720    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq" podStartSLOduration=2.229704688 podStartE2EDuration="2.229704688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:38.228349313 +0000 UTC m=+2494.562926082" watchObservedRunningTime="2026-10-30 22:28:38.229704688 +0000 UTC m=+2494.564281458"
Oct 30 22:28:39.119451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:39.119417    5503 generic.go:356] "Generic (PLEG): container finished" podID="de45cc76-477a-4813-af22-3007a3748e10" containerID="d1855e6e670b933a88524ad28ae7f1b2e4d14cbe4345b4e464e0e05254396a4d" exitCode=0
Oct 30 22:28:39.119926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:39.119496    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq" event={"ID":"de45cc76-477a-4813-af22-3007a3748e10","Type":"ContainerDied","Data":"d1855e6e670b933a88524ad28ae7f1b2e4d14cbe4345b4e464e0e05254396a4d"}
Oct 30 22:28:39.832153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:39.832123    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:40.255428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.255403    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:40.316236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.316209    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de45cc76-477a-4813-af22-3007a3748e10-kube-api-access-k9jxh\" (UniqueName: \"kubernetes.io/projected/de45cc76-477a-4813-af22-3007a3748e10-kube-api-access-k9jxh\") pod \"de45cc76-477a-4813-af22-3007a3748e10\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") "
Oct 30 22:28:40.316362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.316312    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de45cc76-477a-4813-af22-3007a3748e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de45cc76-477a-4813-af22-3007a3748e10-maas-api-xks-ca\") pod \"de45cc76-477a-4813-af22-3007a3748e10\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") "
Oct 30 22:28:40.316491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.316474    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de45cc76-477a-4813-af22-3007a3748e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de45cc76-477a-4813-af22-3007a3748e10-maas-api-openshift-ca\") pod \"de45cc76-477a-4813-af22-3007a3748e10\" (UID: \"de45cc76-477a-4813-af22-3007a3748e10\") "
Oct 30 22:28:40.316848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.316823    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de45cc76-477a-4813-af22-3007a3748e10-maas-api-openshift-ca" pod "de45cc76-477a-4813-af22-3007a3748e10" (UID: "de45cc76-477a-4813-af22-3007a3748e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:40.316978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.316955    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de45cc76-477a-4813-af22-3007a3748e10-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.318517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.318490    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de45cc76-477a-4813-af22-3007a3748e10-maas-api-xks-ca" pod "de45cc76-477a-4813-af22-3007a3748e10" (UID: "de45cc76-477a-4813-af22-3007a3748e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:40.318607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.318551    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de45cc76-477a-4813-af22-3007a3748e10-kube-api-access-k9jxh" pod "de45cc76-477a-4813-af22-3007a3748e10" (UID: "de45cc76-477a-4813-af22-3007a3748e10"). InnerVolumeSpecName "kube-api-access-k9jxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:40.417919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.417879    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jxh\" (UniqueName: \"kubernetes.io/projected/de45cc76-477a-4813-af22-3007a3748e10-kube-api-access-k9jxh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.417919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:40.417914    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de45cc76-477a-4813-af22-3007a3748e10-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.132341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:41.132310    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"
Oct 30 22:28:41.132518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:41.132337    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq" event={"ID":"de45cc76-477a-4813-af22-3007a3748e10","Type":"ContainerDied","Data":"88549266f44cdcca85fd80c90112dfcd5735d61cc73bfbd2dab9e2e62460eb56"}
Oct 30 22:28:41.132518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:41.132393    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88549266f44cdcca85fd80c90112dfcd5735d61cc73bfbd2dab9e2e62460eb56"
Oct 30 22:28:43.848270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.848236    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"]
Oct 30 22:28:43.848968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.848948    5503 cpu_manager.go:414] "containerMap: removing container" podUID="de45cc76-477a-4813-af22-3007a3748e10" containerName="revoke-keys"
Oct 30 22:28:43.849012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.848972    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de45cc76-477a-4813-af22-3007a3748e10" containerName="revoke-keys"
Oct 30 22:28:43.849096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.849085    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="de45cc76-477a-4813-af22-3007a3748e10" containerName="revoke-keys"
Oct 30 22:28:43.852956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.852936    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:43.906090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.906054    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"]
Oct 30 22:28:43.954122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.954087    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:43.954287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.954226    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:28:43.954355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.954285    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:28:43.957052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.957037    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:28:43.958511 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.958492    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:43.958569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.958548    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jq4p\" (UniqueName: \"kubernetes.io/projected/d6d20c3c-40e5-4736-8ba9-4d90e988395a-kube-api-access-8jq4p\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:43.958750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:43.958734    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.059775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.059744    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.059915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.059789    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.059915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.059846    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jq4p\" (UniqueName: \"kubernetes.io/projected/d6d20c3c-40e5-4736-8ba9-4d90e988395a-kube-api-access-8jq4p\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.060358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.060335    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.062176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.062154    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.121822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.121765    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jq4p\" (UniqueName: \"kubernetes.io/projected/d6d20c3c-40e5-4736-8ba9-4d90e988395a-kube-api-access-8jq4p\") pod \"maas-api-revoke-sub-bc553e2d-k5pgs\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.166500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.166468    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:44.404543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.404511    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:44.542862 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:44.542825    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d20c3c_40e5_4736_8ba9_4d90e988395a.slice/crio-40677f78687474a3676a87430ad388b6b8aaa1bd8576eb06230dc27565622c77 WatchSource:0}: Error finding container 40677f78687474a3676a87430ad388b6b8aaa1bd8576eb06230dc27565622c77: Status 404 returned error can't find the container with id 40677f78687474a3676a87430ad388b6b8aaa1bd8576eb06230dc27565622c77
Oct 30 22:28:44.545663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:44.545609    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"]
Oct 30 22:28:45.108027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:45.107948    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:45.161684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:45.161620    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs" event={"ID":"d6d20c3c-40e5-4736-8ba9-4d90e988395a","Type":"ContainerStarted","Data":"0c1ed4d8c72871427675e334b49ae759545c9b50cc414989a79cd61030b6bdca"}
Oct 30 22:28:45.161684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:45.161688    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs" event={"ID":"d6d20c3c-40e5-4736-8ba9-4d90e988395a","Type":"ContainerStarted","Data":"40677f78687474a3676a87430ad388b6b8aaa1bd8576eb06230dc27565622c77"}
Oct 30 22:28:45.281012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:45.280961    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs" podStartSLOduration=2.280948634 podStartE2EDuration="2.280948634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:45.279752421 +0000 UTC m=+2501.614329190" watchObservedRunningTime="2026-10-30 22:28:45.280948634 +0000 UTC m=+2501.615525414"
Oct 30 22:28:46.168935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:46.168901    5503 generic.go:356] "Generic (PLEG): container finished" podID="d6d20c3c-40e5-4736-8ba9-4d90e988395a" containerID="0c1ed4d8c72871427675e334b49ae759545c9b50cc414989a79cd61030b6bdca" exitCode=0
Oct 30 22:28:46.169382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:46.168988    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs" event={"ID":"d6d20c3c-40e5-4736-8ba9-4d90e988395a","Type":"ContainerDied","Data":"0c1ed4d8c72871427675e334b49ae759545c9b50cc414989a79cd61030b6bdca"}
Oct 30 22:28:46.247595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:46.247564    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:28:46.490591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:46.490507    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:47.029528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.029476    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:47.166660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.165006    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"]
Oct 30 22:28:47.184980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.184947    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.264695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.258089    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.265069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.265030    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.265400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.265378    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.265618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.265598    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.266141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.266121    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.266353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.266336    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrrh\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-kube-api-access-pbrrh\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.266547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.266531    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.266772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.266754    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.266958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.266941    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ec8cd4-71b2-4643-a622-f557a2728ca6-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.368715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.367998    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"]
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369292    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369378    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369481    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369533    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369580    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369626    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrrh\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-kube-api-access-pbrrh\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369696    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369774    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.369824    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ec8cd4-71b2-4643-a622-f557a2728ca6-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.371484    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ec8cd4-71b2-4643-a622-f557a2728ca6-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.372741    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.372961    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.403109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.373009    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.404184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.404140    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.408656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.405313    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.412655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.410810    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.529779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.512326    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:47.532829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.532675    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.537032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.537000    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrrh\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-kube-api-access-pbrrh\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:47.584052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.583613    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:47.626297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.626187    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-xks-ca\") pod \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") "
Oct 30 22:28:47.626472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.626300    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d20c3c-40e5-4736-8ba9-4d90e988395a-kube-api-access-8jq4p\" (UniqueName: \"kubernetes.io/projected/d6d20c3c-40e5-4736-8ba9-4d90e988395a-kube-api-access-8jq4p\") pod \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") "
Oct 30 22:28:47.629005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.628966    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-xks-ca" pod "d6d20c3c-40e5-4736-8ba9-4d90e988395a" (UID: "d6d20c3c-40e5-4736-8ba9-4d90e988395a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:47.629166 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.629068    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d20c3c-40e5-4736-8ba9-4d90e988395a-kube-api-access-8jq4p" pod "d6d20c3c-40e5-4736-8ba9-4d90e988395a" (UID: "d6d20c3c-40e5-4736-8ba9-4d90e988395a"). InnerVolumeSpecName "kube-api-access-8jq4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.727420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.727381    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-openshift-ca\") pod \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\" (UID: \"d6d20c3c-40e5-4736-8ba9-4d90e988395a\") "
Oct 30 22:28:47.727892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.727859    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-openshift-ca" pod "d6d20c3c-40e5-4736-8ba9-4d90e988395a" (UID: "d6d20c3c-40e5-4736-8ba9-4d90e988395a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.728357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.728329    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jq4p\" (UniqueName: \"kubernetes.io/projected/d6d20c3c-40e5-4736-8ba9-4d90e988395a-kube-api-access-8jq4p\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.728535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.728515    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.728681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.728664    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d20c3c-40e5-4736-8ba9-4d90e988395a-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.811790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:47.811758    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:48.194981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:48.194952    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"
Oct 30 22:28:48.195368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:48.194976    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs" event={"ID":"d6d20c3c-40e5-4736-8ba9-4d90e988395a","Type":"ContainerDied","Data":"40677f78687474a3676a87430ad388b6b8aaa1bd8576eb06230dc27565622c77"}
Oct 30 22:28:48.195368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:48.195036    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40677f78687474a3676a87430ad388b6b8aaa1bd8576eb06230dc27565622c77"
Oct 30 22:28:48.251970 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:28:48.251939    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ec8cd4_71b2_4643_a622_f557a2728ca6.slice/crio-631289275d4b29e67a7dfea5bfcd46cfbcf0ee7299d6718b5c8a676ef0eff39a WatchSource:0}: Error finding container 631289275d4b29e67a7dfea5bfcd46cfbcf0ee7299d6718b5c8a676ef0eff39a: Status 404 returned error can't find the container with id 631289275d4b29e67a7dfea5bfcd46cfbcf0ee7299d6718b5c8a676ef0eff39a
Oct 30 22:28:48.254529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:48.254493    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:28:48.254651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:48.254576    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:28:48.254651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:48.254619    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:28:48.306644 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:48.306606    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"]
Oct 30 22:28:49.203075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:49.203033    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" event={"ID":"15ec8cd4-71b2-4643-a622-f557a2728ca6","Type":"ContainerStarted","Data":"3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a"}
Oct 30 22:28:49.203574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:49.203095    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" event={"ID":"15ec8cd4-71b2-4643-a622-f557a2728ca6","Type":"ContainerStarted","Data":"631289275d4b29e67a7dfea5bfcd46cfbcf0ee7299d6718b5c8a676ef0eff39a"}
Oct 30 22:28:49.378670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:49.378599    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podStartSLOduration=2.3785844689999998 podStartE2EDuration="2.378584469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:49.376650017 +0000 UTC m=+2505.711226778" watchObservedRunningTime="2026-10-30 22:28:49.378584469 +0000 UTC m=+2505.713161238"
Oct 30 22:28:49.565865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:49.565787    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:49.812528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:49.812497    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:49.814373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:49.814349    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:49.814525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:49.814396    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:50.812542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:50.812507    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:50.812937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:50.812567    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:51.812872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:51.812837    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:51.813240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:51.812886    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:51.886233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:51.886198    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:52.098431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:52.098349    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:52.812805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:52.812769    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:52.812980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:52.812828    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:53.812173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:53.812137    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:53.812364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:53.812196    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:54.701354 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:54.701317    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"]
Oct 30 22:28:54.701891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:54.701532    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" containerID="cri-o://3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64" gracePeriod=30
Oct 30 22:28:54.789154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:54.789112    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:54.813007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:54.812979    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:28:54.813196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:54.813031    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:28:55.203344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:55.203297    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:55.386170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:55.386129    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:55.817611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:55.817579    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:55.818081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:55.817875    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:55.818574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:55.818557    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:28:58.101823 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:28:58.101782    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:28:59.330931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:59.330894    5503 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6f54779779-svxzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:28:59.331308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:59.330965    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:28:59.878927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:59.878894    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:28:59.971100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:28:59.971041    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:29:00.082660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082601    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b58ca3-706f-4c30-9608-d0cbf626e912-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b58ca3-706f-4c30-9608-d0cbf626e912-istiod-ca-cert\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.082858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082754    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-credential-socket\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.082858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082778    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-socket\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.082858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082797    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-data\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.082858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082825    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-token\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-token\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.083105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082867    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-kube-api-access-qbh55\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-kube-api-access-qbh55\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.083105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082919    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-podinfo\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.083105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.082990    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-envoy\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.083105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083039    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-certs\") pod \"79b58ca3-706f-4c30-9608-d0cbf626e912\" (UID: \"79b58ca3-706f-4c30-9608-d0cbf626e912\") "
Oct 30 22:29:00.083105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083038    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b58ca3-706f-4c30-9608-d0cbf626e912-istiod-ca-cert" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:00.083350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083105    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-socket" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:00.083350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083125    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-credential-socket" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:00.083350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083142    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-data" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:00.083350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083267    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-certs" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:00.083629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083609    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.083629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083654    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b58ca3-706f-4c30-9608-d0cbf626e912-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.083775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083671    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.083775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083685    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.083775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.083698    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.085739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.085711    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-token" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:00.085850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.085746    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-envoy" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:00.085850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.085752    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-podinfo" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:00.085850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.085772    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-kube-api-access-qbh55" pod "79b58ca3-706f-4c30-9608-d0cbf626e912" (UID: "79b58ca3-706f-4c30-9608-d0cbf626e912"). InnerVolumeSpecName "kube-api-access-qbh55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:00.184317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.184283    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbh55\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-kube-api-access-qbh55\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.184317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.184315    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.184520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.184329    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.184520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.184342    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b58ca3-706f-4c30-9608-d0cbf626e912-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:00.298836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.298750    5503 generic.go:356] "Generic (PLEG): container finished" podID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerID="3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64" exitCode=0
Oct 30 22:29:00.298836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.298802    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" event={"ID":"79b58ca3-706f-4c30-9608-d0cbf626e912","Type":"ContainerDied","Data":"3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64"}
Oct 30 22:29:00.298836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.298809    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"
Oct 30 22:29:00.299058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.298855    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp" event={"ID":"79b58ca3-706f-4c30-9608-d0cbf626e912","Type":"ContainerDied","Data":"56be2aa23a6b2957605f6a2531c1c91f07e7a330136a25f6d9c01cbb6124e258"}
Oct 30 22:29:00.299058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.298887    5503 scope.go:122] "RemoveContainer" containerID="3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64"
Oct 30 22:29:00.309909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.309890    5503 scope.go:122] "RemoveContainer" containerID="3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64"
Oct 30 22:29:00.310160 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:29:00.310136    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64\": container with ID starting with 3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64 not found: ID does not exist" containerID="3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64"
Oct 30 22:29:00.310234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.310172    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64"} err="failed to get container status \"3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64\": rpc error: code = NotFound desc = could not find container \"3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64\": container with ID starting with 3f8033a976d4ddc30d1f0ea22ff4a0c4933fcbf9c1d9a25abdbe28e292f43c64 not found: ID does not exist"
Oct 30 22:29:00.470100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.470059    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"]
Oct 30 22:29:00.473087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:00.473063    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-svxzp"]
Oct 30 22:29:01.788149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:01.788114    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" path="/var/lib/kubelet/pods/79b58ca3-706f-4c30-9608-d0cbf626e912/volumes"
Oct 30 22:29:02.279047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:02.279014    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"]
Oct 30 22:29:02.279308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:02.279256    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" containerID="cri-o://b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537" gracePeriod=30
Oct 30 22:29:03.025817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.025783    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"]
Oct 30 22:29:03.026252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.026234    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d6d20c3c-40e5-4736-8ba9-4d90e988395a" containerName="revoke-keys"
Oct 30 22:29:03.026302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.026255    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d20c3c-40e5-4736-8ba9-4d90e988395a" containerName="revoke-keys"
Oct 30 22:29:03.026302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.026291    5503 cpu_manager.go:414] "containerMap: removing container" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy"
Oct 30 22:29:03.026302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.026296    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy"
Oct 30 22:29:03.026397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.026377    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d20c3c-40e5-4736-8ba9-4d90e988395a" containerName="revoke-keys"
Oct 30 22:29:03.026397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.026393    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b58ca3-706f-4c30-9608-d0cbf626e912" containerName="istio-proxy"
Oct 30 22:29:03.030899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.030884    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.084067 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.084035    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"]
Oct 30 22:29:03.117180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.117149    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.117341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.117199    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.117399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.117336    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chzw\" (UniqueName: \"kubernetes.io/projected/682118b1-1dbb-4422-9ad3-7af795df22d9-kube-api-access-5chzw\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.131893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.131858    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:03.131893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.131867    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:29:03.132070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.131899    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:03.135220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.135205    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:03.218138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.218107    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.218297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.218170    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chzw\" (UniqueName: \"kubernetes.io/projected/682118b1-1dbb-4422-9ad3-7af795df22d9-kube-api-access-5chzw\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.218297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.218258    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.218844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.218821    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.220427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.220400    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.280306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.280187    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:03.280597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.280577    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chzw\" (UniqueName: \"kubernetes.io/projected/682118b1-1dbb-4422-9ad3-7af795df22d9-kube-api-access-5chzw\") pod \"maas-api-revoke-sub-92ac6806-r4xwk\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.344246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.344205    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:03.714845 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:03.714816    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682118b1_1dbb_4422_9ad3_7af795df22d9.slice/crio-27bf260f3b64c01d2b8941e074f86c1692b1c6293fdd60a42a067e0c09c3e8b7 WatchSource:0}: Error finding container 27bf260f3b64c01d2b8941e074f86c1692b1c6293fdd60a42a067e0c09c3e8b7: Status 404 returned error can't find the container with id 27bf260f3b64c01d2b8941e074f86c1692b1c6293fdd60a42a067e0c09c3e8b7
Oct 30 22:29:03.717870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:03.717845    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"]
Oct 30 22:29:04.164746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:04.164717    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:04.334703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:04.334593    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk" event={"ID":"682118b1-1dbb-4422-9ad3-7af795df22d9","Type":"ContainerStarted","Data":"a4530251cbfd51f20ac7ebc2f30a12f672eed6f82037f240aa19b29f0c21d7ff"}
Oct 30 22:29:04.334703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:04.334676    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk" event={"ID":"682118b1-1dbb-4422-9ad3-7af795df22d9","Type":"ContainerStarted","Data":"27bf260f3b64c01d2b8941e074f86c1692b1c6293fdd60a42a067e0c09c3e8b7"}
Oct 30 22:29:04.502081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:04.502026    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:05.003919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:05.003843    5503 scope.go:122] "RemoveContainer" containerID="9e8dad31733071808d727fd09a7683ab2eda6221e47e1ba1a457db2e510be063"
Oct 30 22:29:05.141362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:05.141262    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:05.360710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:05.360605    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:05.363513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:05.363472    5503 generic.go:356] "Generic (PLEG): container finished" podID="682118b1-1dbb-4422-9ad3-7af795df22d9" containerID="a4530251cbfd51f20ac7ebc2f30a12f672eed6f82037f240aa19b29f0c21d7ff" exitCode=0
Oct 30 22:29:05.363816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:05.363784    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk" event={"ID":"682118b1-1dbb-4422-9ad3-7af795df22d9","Type":"ContainerDied","Data":"a4530251cbfd51f20ac7ebc2f30a12f672eed6f82037f240aa19b29f0c21d7ff"}
Oct 30 22:29:05.655168 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:05.655133    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:05.853872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:05.853840    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:06.012933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.012848    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:06.107339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.107299    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"]
Oct 30 22:29:06.158444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.158392    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cf5019-nbxzr"]
Oct 30 22:29:06.512423 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.512400    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:06.587689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.587630    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:06.679743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.679699    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682118b1-1dbb-4422-9ad3-7af795df22d9-kube-api-access-5chzw\" (UniqueName: \"kubernetes.io/projected/682118b1-1dbb-4422-9ad3-7af795df22d9-kube-api-access-5chzw\") pod \"682118b1-1dbb-4422-9ad3-7af795df22d9\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") "
Oct 30 22:29:06.680757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.680733    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-openshift-ca\") pod \"682118b1-1dbb-4422-9ad3-7af795df22d9\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") "
Oct 30 22:29:06.680978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.680959    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-xks-ca\") pod \"682118b1-1dbb-4422-9ad3-7af795df22d9\" (UID: \"682118b1-1dbb-4422-9ad3-7af795df22d9\") "
Oct 30 22:29:06.682136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.682110    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-openshift-ca" pod "682118b1-1dbb-4422-9ad3-7af795df22d9" (UID: "682118b1-1dbb-4422-9ad3-7af795df22d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:06.688867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.688838    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-xks-ca" pod "682118b1-1dbb-4422-9ad3-7af795df22d9" (UID: "682118b1-1dbb-4422-9ad3-7af795df22d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:06.689307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.689265    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682118b1-1dbb-4422-9ad3-7af795df22d9-kube-api-access-5chzw" pod "682118b1-1dbb-4422-9ad3-7af795df22d9" (UID: "682118b1-1dbb-4422-9ad3-7af795df22d9"). InnerVolumeSpecName "kube-api-access-5chzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:06.783229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.783194    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chzw\" (UniqueName: \"kubernetes.io/projected/682118b1-1dbb-4422-9ad3-7af795df22d9-kube-api-access-5chzw\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:06.783229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.783231    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:06.783436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:06.783249    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682118b1-1dbb-4422-9ad3-7af795df22d9-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.021793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.021710    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:29:07.021793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.021780    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:29:07.377715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.377685    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"
Oct 30 22:29:07.377715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.377700    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk" event={"ID":"682118b1-1dbb-4422-9ad3-7af795df22d9","Type":"ContainerDied","Data":"27bf260f3b64c01d2b8941e074f86c1692b1c6293fdd60a42a067e0c09c3e8b7"}
Oct 30 22:29:07.377941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.377753    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27bf260f3b64c01d2b8941e074f86c1692b1c6293fdd60a42a067e0c09c3e8b7"
Oct 30 22:29:07.542776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.542749    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:29:07.693499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693459    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-certs\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.693700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693525    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-data\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.693700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693578    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-podinfo\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.693700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693669    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-kube-api-access-jgnlx\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-kube-api-access-jgnlx\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.693898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693716    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-envoy\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.693898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693745    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f694c451-7d40-4d19-a63f-4d62c52c734b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694c451-7d40-4d19-a63f-4d62c52c734b-istiod-ca-cert\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.693898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693835    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-certs" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:07.693898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693851    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-data" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:07.693898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693848    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-credential-socket\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.694155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693905    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-token\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-token\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.694155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.693968    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-socket\") pod \"f694c451-7d40-4d19-a63f-4d62c52c734b\" (UID: \"f694c451-7d40-4d19-a63f-4d62c52c734b\") "
Oct 30 22:29:07.694155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694094    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-credential-socket" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:07.694318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694205    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f694c451-7d40-4d19-a63f-4d62c52c734b-istiod-ca-cert" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:07.694388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694366    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-socket" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:07.694481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694458    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f694c451-7d40-4d19-a63f-4d62c52c734b-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.694544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694486    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.694544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694500    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.694544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694513    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.694544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.694526    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.696156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.696131    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-kube-api-access-jgnlx" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "kube-api-access-jgnlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:07.696294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.696190    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-envoy" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:07.696375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.696345    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-token" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:07.696375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.696347    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-podinfo" pod "f694c451-7d40-4d19-a63f-4d62c52c734b" (UID: "f694c451-7d40-4d19-a63f-4d62c52c734b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:07.788154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.788114    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d931c45b-62ed-41b4-a59b-e39e62752027" path="/var/lib/kubelet/pods/d931c45b-62ed-41b4-a59b-e39e62752027/volumes"
Oct 30 22:29:07.795955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.795928    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.795955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.795954    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgnlx\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-kube-api-access-jgnlx\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.796159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.795970    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:07.796159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:07.795985    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f694c451-7d40-4d19-a63f-4d62c52c734b-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:08.385601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.385565    5503 generic.go:356] "Generic (PLEG): container finished" podID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerID="b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537" exitCode=0
Oct 30 22:29:08.385785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.385611    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"
Oct 30 22:29:08.385785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.385668    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" event={"ID":"f694c451-7d40-4d19-a63f-4d62c52c734b","Type":"ContainerDied","Data":"b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537"}
Oct 30 22:29:08.385785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.385724    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54" event={"ID":"f694c451-7d40-4d19-a63f-4d62c52c734b","Type":"ContainerDied","Data":"fb967caad6bffec6d851670ad9f3c11e5248adda2b79ca208f659037b49f82fd"}
Oct 30 22:29:08.385785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.385745    5503 scope.go:122] "RemoveContainer" containerID="b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537"
Oct 30 22:29:08.395752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.395723    5503 scope.go:122] "RemoveContainer" containerID="b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537"
Oct 30 22:29:08.396029 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:29:08.396008    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537\": container with ID starting with b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537 not found: ID does not exist" containerID="b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537"
Oct 30 22:29:08.396089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.396037    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537"} err="failed to get container status \"b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537\": rpc error: code = NotFound desc = could not find container \"b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537\": container with ID starting with b3e62c1518d05296dbe564036f066ca5567bd2913d6d876f91880c22809d8537 not found: ID does not exist"
Oct 30 22:29:08.558061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.558021    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"]
Oct 30 22:29:08.611720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:08.611682    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-ddr54"]
Oct 30 22:29:09.364479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:09.364444    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:09.539408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:09.539372    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:09.697858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:09.697826    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:09.788315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:09.788284    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" path="/var/lib/kubelet/pods/f694c451-7d40-4d19-a63f-4d62c52c734b/volumes"
Oct 30 22:29:10.903723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:10.903691    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:12.641155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:12.641118    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:12.821897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:12.821860    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:13.698602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:13.698571    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:13.876657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:13.876085    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:14.086382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:14.086287    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:15.944690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:15.944658    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:17.620182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.620144    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"]
Oct 30 22:29:17.622470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.620555    5503 cpu_manager.go:414] "containerMap: removing container" podUID="682118b1-1dbb-4422-9ad3-7af795df22d9" containerName="revoke-keys"
Oct 30 22:29:17.622470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.620566    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682118b1-1dbb-4422-9ad3-7af795df22d9" containerName="revoke-keys"
Oct 30 22:29:17.622470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.620580    5503 cpu_manager.go:414] "containerMap: removing container" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy"
Oct 30 22:29:17.622470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.620585    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy"
Oct 30 22:29:17.622470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.620680    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="682118b1-1dbb-4422-9ad3-7af795df22d9" containerName="revoke-keys"
Oct 30 22:29:17.622470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.620691    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f694c451-7d40-4d19-a63f-4d62c52c734b" containerName="istio-proxy"
Oct 30 22:29:17.623744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.623727    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.677346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.677313    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:17.677518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.677482    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:29:17.678783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.678757    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"]
Oct 30 22:29:17.714714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.714678    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.714904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.714724    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.714904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.714840    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92vm\" (UniqueName: \"kubernetes.io/projected/77c724cf-9ad6-4774-9054-6bc794a174ce-kube-api-access-w92vm\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.726477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.726445    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:17.726477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.726451    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:17.815502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.815467    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.815502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.815502    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.815816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.815536    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92vm\" (UniqueName: \"kubernetes.io/projected/77c724cf-9ad6-4774-9054-6bc794a174ce-kube-api-access-w92vm\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.816278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.816257    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.818153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.818134    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.877914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.877847    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92vm\" (UniqueName: \"kubernetes.io/projected/77c724cf-9ad6-4774-9054-6bc794a174ce-kube-api-access-w92vm\") pod \"maas-api-revoke-sub-71a55a66-gl96s\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:17.936426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:17.936389    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:18.306090 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:18.306057    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c724cf_9ad6_4774_9054_6bc794a174ce.slice/crio-9e0865e357e17a6a93d9879fde9e73aa1e4c9ea9e0b8553e1de4b69029b93c70 WatchSource:0}: Error finding container 9e0865e357e17a6a93d9879fde9e73aa1e4c9ea9e0b8553e1de4b69029b93c70: Status 404 returned error can't find the container with id 9e0865e357e17a6a93d9879fde9e73aa1e4c9ea9e0b8553e1de4b69029b93c70
Oct 30 22:29:18.309886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.309856    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"]
Oct 30 22:29:18.343934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.343911    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"]
Oct 30 22:29:18.348913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.348891    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.413257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.413228    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"]
Oct 30 22:29:18.420682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.420650    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.420831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.420745    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-787mz\" (UniqueName: \"kubernetes.io/projected/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-kube-api-access-787mz\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.420831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.420778    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.470030 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.469994    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s" event={"ID":"77c724cf-9ad6-4774-9054-6bc794a174ce","Type":"ContainerStarted","Data":"9e0865e357e17a6a93d9879fde9e73aa1e4c9ea9e0b8553e1de4b69029b93c70"}
Oct 30 22:29:18.521976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.521945    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.522101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.522032    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-787mz\" (UniqueName: \"kubernetes.io/projected/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-kube-api-access-787mz\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.522101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.522065    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.522735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.522709    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.524305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.524277    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.584368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.584309    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-787mz\" (UniqueName: \"kubernetes.io/projected/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-kube-api-access-787mz\") pod \"maas-api-revoke-sub-8cc3b91a-znzzf\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:18.662367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:18.662333    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:19.037342 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:19.037313    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14dc26a_269c_4bdb_aaeb_8546c5a460e7.slice/crio-addcae17170257dbc26f34db03fa9c6c5edbe516f9cd1be0baa4168d12695fc1 WatchSource:0}: Error finding container addcae17170257dbc26f34db03fa9c6c5edbe516f9cd1be0baa4168d12695fc1: Status 404 returned error can't find the container with id addcae17170257dbc26f34db03fa9c6c5edbe516f9cd1be0baa4168d12695fc1
Oct 30 22:29:19.039690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.039663    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"]
Oct 30 22:29:19.237184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.237153    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"]
Oct 30 22:29:19.240825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.240805    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.297377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.297310    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"]
Oct 30 22:29:19.327914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.327890    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.328044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.328027    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jl2\" (UniqueName: \"kubernetes.io/projected/09b55996-e3a2-4367-a9cc-5d2e5c90e324-kube-api-access-r9jl2\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.328137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.328066    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.429263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.429234    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jl2\" (UniqueName: \"kubernetes.io/projected/09b55996-e3a2-4367-a9cc-5d2e5c90e324-kube-api-access-r9jl2\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.429411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.429276    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.429411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.429340    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.429952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.429931    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.431590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.431562    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.477607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.477576    5503 generic.go:356] "Generic (PLEG): container finished" podID="77c724cf-9ad6-4774-9054-6bc794a174ce" containerID="a9820c397bb31b58cf54802e1fd0d6a8d8a4cbf8dfe27fb88b949d3c17820798" exitCode=0
Oct 30 22:29:19.477726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.477627    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s" event={"ID":"77c724cf-9ad6-4774-9054-6bc794a174ce","Type":"ContainerDied","Data":"a9820c397bb31b58cf54802e1fd0d6a8d8a4cbf8dfe27fb88b949d3c17820798"}
Oct 30 22:29:19.479448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.479428    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf" event={"ID":"f14dc26a-269c-4bdb-aaeb-8546c5a460e7","Type":"ContainerStarted","Data":"8be6ff6979c3b6da0c0a1cb3f4655321256e7f02834aaa8cbb747b9334b9b2d8"}
Oct 30 22:29:19.479534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.479460    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf" event={"ID":"f14dc26a-269c-4bdb-aaeb-8546c5a460e7","Type":"ContainerStarted","Data":"addcae17170257dbc26f34db03fa9c6c5edbe516f9cd1be0baa4168d12695fc1"}
Oct 30 22:29:19.491407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.491385    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jl2\" (UniqueName: \"kubernetes.io/projected/09b55996-e3a2-4367-a9cc-5d2e5c90e324-kube-api-access-r9jl2\") pod \"maas-api-revoke-sub-9a68cbd2-4pb7f\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.623053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.622979    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:19.723234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:19.723169    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf" podStartSLOduration=1.723150468 podStartE2EDuration="1.723150468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:19.72170979 +0000 UTC m=+2536.056286558" watchObservedRunningTime="2026-10-30 22:29:19.723150468 +0000 UTC m=+2536.057727239"
Oct 30 22:29:20.009053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.009029    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"]
Oct 30 22:29:20.065659 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:20.065605    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b55996_e3a2_4367_a9cc_5d2e5c90e324.slice/crio-1cab32e9a6ddcc446134a94579c9c234772eb8c68f53bff8aed2fb06ccf45d16 WatchSource:0}: Error finding container 1cab32e9a6ddcc446134a94579c9c234772eb8c68f53bff8aed2fb06ccf45d16: Status 404 returned error can't find the container with id 1cab32e9a6ddcc446134a94579c9c234772eb8c68f53bff8aed2fb06ccf45d16
Oct 30 22:29:20.487972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.487931    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f" event={"ID":"09b55996-e3a2-4367-a9cc-5d2e5c90e324","Type":"ContainerStarted","Data":"a9c12809d31d2fc252f0b44ca87dce2bb3bb813c21c5b5b127e0ef3f38bc8906"}
Oct 30 22:29:20.488126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.487994    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f" event={"ID":"09b55996-e3a2-4367-a9cc-5d2e5c90e324","Type":"ContainerStarted","Data":"1cab32e9a6ddcc446134a94579c9c234772eb8c68f53bff8aed2fb06ccf45d16"}
Oct 30 22:29:20.489517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.489489    5503 generic.go:356] "Generic (PLEG): container finished" podID="f14dc26a-269c-4bdb-aaeb-8546c5a460e7" containerID="8be6ff6979c3b6da0c0a1cb3f4655321256e7f02834aaa8cbb747b9334b9b2d8" exitCode=0
Oct 30 22:29:20.489659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.489566    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf" event={"ID":"f14dc26a-269c-4bdb-aaeb-8546c5a460e7","Type":"ContainerDied","Data":"8be6ff6979c3b6da0c0a1cb3f4655321256e7f02834aaa8cbb747b9334b9b2d8"}
Oct 30 22:29:20.607176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.607128    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f" podStartSLOduration=1.607110296 podStartE2EDuration="1.607110296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:20.605133732 +0000 UTC m=+2536.939710502" watchObservedRunningTime="2026-10-30 22:29:20.607110296 +0000 UTC m=+2536.941687095"
Oct 30 22:29:20.627370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.627344    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:20.644029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.644006    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-xks-ca\") pod \"77c724cf-9ad6-4774-9054-6bc794a174ce\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") "
Oct 30 22:29:20.644142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.644127    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c724cf-9ad6-4774-9054-6bc794a174ce-kube-api-access-w92vm\" (UniqueName: \"kubernetes.io/projected/77c724cf-9ad6-4774-9054-6bc794a174ce-kube-api-access-w92vm\") pod \"77c724cf-9ad6-4774-9054-6bc794a174ce\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") "
Oct 30 22:29:20.644237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.644222    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-openshift-ca\") pod \"77c724cf-9ad6-4774-9054-6bc794a174ce\" (UID: \"77c724cf-9ad6-4774-9054-6bc794a174ce\") "
Oct 30 22:29:20.644559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.644532    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-openshift-ca" pod "77c724cf-9ad6-4774-9054-6bc794a174ce" (UID: "77c724cf-9ad6-4774-9054-6bc794a174ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:20.644701 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.644622    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:20.646100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.646074    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-xks-ca" pod "77c724cf-9ad6-4774-9054-6bc794a174ce" (UID: "77c724cf-9ad6-4774-9054-6bc794a174ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:20.646200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.646148    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c724cf-9ad6-4774-9054-6bc794a174ce-kube-api-access-w92vm" pod "77c724cf-9ad6-4774-9054-6bc794a174ce" (UID: "77c724cf-9ad6-4774-9054-6bc794a174ce"). InnerVolumeSpecName "kube-api-access-w92vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:20.742124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.742036    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:20.745321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.745292    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c724cf-9ad6-4774-9054-6bc794a174ce-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:20.745380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:20.745325    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w92vm\" (UniqueName: \"kubernetes.io/projected/77c724cf-9ad6-4774-9054-6bc794a174ce-kube-api-access-w92vm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:21.496836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.496804    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"
Oct 30 22:29:21.497002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.496837    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s" event={"ID":"77c724cf-9ad6-4774-9054-6bc794a174ce","Type":"ContainerDied","Data":"9e0865e357e17a6a93d9879fde9e73aa1e4c9ea9e0b8553e1de4b69029b93c70"}
Oct 30 22:29:21.497002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.496891    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e0865e357e17a6a93d9879fde9e73aa1e4c9ea9e0b8553e1de4b69029b93c70"
Oct 30 22:29:21.498475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.498449    5503 generic.go:356] "Generic (PLEG): container finished" podID="09b55996-e3a2-4367-a9cc-5d2e5c90e324" containerID="a9c12809d31d2fc252f0b44ca87dce2bb3bb813c21c5b5b127e0ef3f38bc8906" exitCode=0
Oct 30 22:29:21.498621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.498540    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f" event={"ID":"09b55996-e3a2-4367-a9cc-5d2e5c90e324","Type":"ContainerDied","Data":"a9c12809d31d2fc252f0b44ca87dce2bb3bb813c21c5b5b127e0ef3f38bc8906"}
Oct 30 22:29:21.649329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.649303    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:21.668246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.668216    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-xks-ca\") pod \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") "
Oct 30 22:29:21.668434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.668314    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-kube-api-access-787mz\" (UniqueName: \"kubernetes.io/projected/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-kube-api-access-787mz\") pod \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") "
Oct 30 22:29:21.668434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.668424    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-openshift-ca\") pod \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\" (UID: \"f14dc26a-269c-4bdb-aaeb-8546c5a460e7\") "
Oct 30 22:29:21.668833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.668806    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-openshift-ca" pod "f14dc26a-269c-4bdb-aaeb-8546c5a460e7" (UID: "f14dc26a-269c-4bdb-aaeb-8546c5a460e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:21.670387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.670359    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-xks-ca" pod "f14dc26a-269c-4bdb-aaeb-8546c5a460e7" (UID: "f14dc26a-269c-4bdb-aaeb-8546c5a460e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:21.670483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.670421    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-kube-api-access-787mz" pod "f14dc26a-269c-4bdb-aaeb-8546c5a460e7" (UID: "f14dc26a-269c-4bdb-aaeb-8546c5a460e7"). InnerVolumeSpecName "kube-api-access-787mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:21.770285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.770201    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:21.770285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.770237    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:21.770285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:21.770253    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-787mz\" (UniqueName: \"kubernetes.io/projected/f14dc26a-269c-4bdb-aaeb-8546c5a460e7-kube-api-access-787mz\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:22.506228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.506197    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"
Oct 30 22:29:22.506404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.506226    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf" event={"ID":"f14dc26a-269c-4bdb-aaeb-8546c5a460e7","Type":"ContainerDied","Data":"addcae17170257dbc26f34db03fa9c6c5edbe516f9cd1be0baa4168d12695fc1"}
Oct 30 22:29:22.506404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.506275    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="addcae17170257dbc26f34db03fa9c6c5edbe516f9cd1be0baa4168d12695fc1"
Oct 30 22:29:22.652014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.651987    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:22.678409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.678377    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-xks-ca\") pod \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") "
Oct 30 22:29:22.678590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.678478    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-openshift-ca\") pod \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") "
Oct 30 22:29:22.678590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.678523    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b55996-e3a2-4367-a9cc-5d2e5c90e324-kube-api-access-r9jl2\" (UniqueName: \"kubernetes.io/projected/09b55996-e3a2-4367-a9cc-5d2e5c90e324-kube-api-access-r9jl2\") pod \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\" (UID: \"09b55996-e3a2-4367-a9cc-5d2e5c90e324\") "
Oct 30 22:29:22.678912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.678882    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-openshift-ca" pod "09b55996-e3a2-4367-a9cc-5d2e5c90e324" (UID: "09b55996-e3a2-4367-a9cc-5d2e5c90e324"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:22.680568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.680541    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-xks-ca" pod "09b55996-e3a2-4367-a9cc-5d2e5c90e324" (UID: "09b55996-e3a2-4367-a9cc-5d2e5c90e324"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:22.680691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.680594    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b55996-e3a2-4367-a9cc-5d2e5c90e324-kube-api-access-r9jl2" pod "09b55996-e3a2-4367-a9cc-5d2e5c90e324" (UID: "09b55996-e3a2-4367-a9cc-5d2e5c90e324"). InnerVolumeSpecName "kube-api-access-r9jl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:22.780407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.780332    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9jl2\" (UniqueName: \"kubernetes.io/projected/09b55996-e3a2-4367-a9cc-5d2e5c90e324-kube-api-access-r9jl2\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:22.780407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.780369    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:22.780407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:22.780384    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09b55996-e3a2-4367-a9cc-5d2e5c90e324-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:23.513115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:23.513083    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"
Oct 30 22:29:23.513456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:23.513117    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f" event={"ID":"09b55996-e3a2-4367-a9cc-5d2e5c90e324","Type":"ContainerDied","Data":"1cab32e9a6ddcc446134a94579c9c234772eb8c68f53bff8aed2fb06ccf45d16"}
Oct 30 22:29:23.513456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:23.513169    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cab32e9a6ddcc446134a94579c9c234772eb8c68f53bff8aed2fb06ccf45d16"
Oct 30 22:29:24.072857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:24.072821    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:25.227134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:25.227094    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:25.381817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:25.381784    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:28.099878 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:29:28.099840    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:29:28.580783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.580754    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"]
Oct 30 22:29:28.581178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581166    5503 cpu_manager.go:414] "containerMap: removing container" podUID="f14dc26a-269c-4bdb-aaeb-8546c5a460e7" containerName="revoke-keys"
Oct 30 22:29:28.581224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581179    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14dc26a-269c-4bdb-aaeb-8546c5a460e7" containerName="revoke-keys"
Oct 30 22:29:28.581224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581204    5503 cpu_manager.go:414] "containerMap: removing container" podUID="77c724cf-9ad6-4774-9054-6bc794a174ce" containerName="revoke-keys"
Oct 30 22:29:28.581224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581212    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c724cf-9ad6-4774-9054-6bc794a174ce" containerName="revoke-keys"
Oct 30 22:29:28.581224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581224    5503 cpu_manager.go:414] "containerMap: removing container" podUID="09b55996-e3a2-4367-a9cc-5d2e5c90e324" containerName="revoke-keys"
Oct 30 22:29:28.581368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581229    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b55996-e3a2-4367-a9cc-5d2e5c90e324" containerName="revoke-keys"
Oct 30 22:29:28.581368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581290    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b55996-e3a2-4367-a9cc-5d2e5c90e324" containerName="revoke-keys"
Oct 30 22:29:28.581368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581311    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14dc26a-269c-4bdb-aaeb-8546c5a460e7" containerName="revoke-keys"
Oct 30 22:29:28.581368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.581321    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c724cf-9ad6-4774-9054-6bc794a174ce" containerName="revoke-keys"
Oct 30 22:29:28.584794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.584778    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.642767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.642729    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.642969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.642785    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.642969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.642894    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47233975-6225-4d8e-a419-e203fc752e15-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.642969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.642937    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.643155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.643065    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.643155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.643094    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.643155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.643119    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94glv\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-kube-api-access-94glv\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.643322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.643207    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.643322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.643259    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47233975-6225-4d8e-a419-e203fc752e15-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.644019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.643993    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"]
Oct 30 22:29:28.744559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.744523    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.744764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.744585    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.744764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.744630    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47233975-6225-4d8e-a419-e203fc752e15-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.744849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.744784    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.744916 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.744900    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.744959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.744935    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.744965    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94glv\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-kube-api-access-94glv\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.745005    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.745011    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.745047    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47233975-6225-4d8e-a419-e203fc752e15-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.745120    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.745365    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.745394    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47233975-6225-4d8e-a419-e203fc752e15-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.745697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.745673    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.747046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.747022    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.747523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.747504    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47233975-6225-4d8e-a419-e203fc752e15-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.807863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.807826    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94glv\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-kube-api-access-94glv\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.808038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.807882    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:28.899401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:28.899313    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:29.321913 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:29.321882    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47233975_6225_4d8e_a419_e203fc752e15.slice/crio-39ebb6fe45243c3ed844992d0ccb2c6a11ac049c1a6600e0675607ca455a58ff WatchSource:0}: Error finding container 39ebb6fe45243c3ed844992d0ccb2c6a11ac049c1a6600e0675607ca455a58ff: Status 404 returned error can't find the container with id 39ebb6fe45243c3ed844992d0ccb2c6a11ac049c1a6600e0675607ca455a58ff
Oct 30 22:29:29.324496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.324459    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:29:29.324605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.324529    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:29:29.324605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.324560    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:29:29.325201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.325176    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"]
Oct 30 22:29:29.560127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.560093    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" event={"ID":"47233975-6225-4d8e-a419-e203fc752e15","Type":"ContainerStarted","Data":"c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b"}
Oct 30 22:29:29.560312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.560148    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" event={"ID":"47233975-6225-4d8e-a419-e203fc752e15","Type":"ContainerStarted","Data":"39ebb6fe45243c3ed844992d0ccb2c6a11ac049c1a6600e0675607ca455a58ff"}
Oct 30 22:29:29.687281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.687228    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podStartSLOduration=1.6872134239999999 podStartE2EDuration="1.687213424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:29.684333059 +0000 UTC m=+2546.018909828" watchObservedRunningTime="2026-10-30 22:29:29.687213424 +0000 UTC m=+2546.021790193"
Oct 30 22:29:29.900380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.900345    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:29.940513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:29.940407    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:30.901172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:30.901133    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:29:30.901663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:30.901196    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:29:31.899900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:31.899866    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:29:31.900055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:31.899917    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:29:32.900375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:32.900338    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:29:32.900977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:32.900944    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:29:32.978177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:32.978143    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"]
Oct 30 22:29:32.981816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:32.981800    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.043707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.043670    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"]
Oct 30 22:29:33.090970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.090943    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:33.091170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.090943    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:29:33.091293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.090953    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:33.091293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.091060    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:33.102604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.102577    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.102849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.102822    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whdm\" (UniqueName: \"kubernetes.io/projected/4c9bf3be-be04-4b49-b8fe-ba65430e9925-kube-api-access-6whdm\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.102949 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.102869    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.204055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.203979    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.204207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.204128    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whdm\" (UniqueName: \"kubernetes.io/projected/4c9bf3be-be04-4b49-b8fe-ba65430e9925-kube-api-access-6whdm\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.204207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.204158    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.204709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.204681    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.206409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.206385    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.267558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.267526    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whdm\" (UniqueName: \"kubernetes.io/projected/4c9bf3be-be04-4b49-b8fe-ba65430e9925-kube-api-access-6whdm\") pod \"maas-api-revoke-sub-6cb0cb8b-dhg5h\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.295824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.295799    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:33.736660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.715492    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"]
Oct 30 22:29:33.900235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.900200    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:29:33.900409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:33.900284    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:29:34.351202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.351093    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.507768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.507727    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.605498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.605416    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h" event={"ID":"4c9bf3be-be04-4b49-b8fe-ba65430e9925","Type":"ContainerStarted","Data":"1098db3fcd1e147350df394b71a1784b9ff824c644461ee1a083474a5fc7b8ca"}
Oct 30 22:29:34.605794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.605768    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h" event={"ID":"4c9bf3be-be04-4b49-b8fe-ba65430e9925","Type":"ContainerStarted","Data":"332a4bc663d029852ebecefd7cda11d36820aec05e5b7ccda61a7804f3acba04"}
Oct 30 22:29:34.670102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.670065    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:34.735156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.735056    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h" podStartSLOduration=2.735037506 podStartE2EDuration="2.735037506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:34.731069271 +0000 UTC m=+2551.065646041" watchObservedRunningTime="2026-10-30 22:29:34.735037506 +0000 UTC m=+2551.069614276"
Oct 30 22:29:34.900080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.899984    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:29:34.900080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:34.900042    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:29:35.615150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:35.615111    5503 generic.go:356] "Generic (PLEG): container finished" podID="4c9bf3be-be04-4b49-b8fe-ba65430e9925" containerID="1098db3fcd1e147350df394b71a1784b9ff824c644461ee1a083474a5fc7b8ca" exitCode=0
Oct 30 22:29:35.615582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:35.615192    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h" event={"ID":"4c9bf3be-be04-4b49-b8fe-ba65430e9925","Type":"ContainerDied","Data":"1098db3fcd1e147350df394b71a1784b9ff824c644461ee1a083474a5fc7b8ca"}
Oct 30 22:29:35.899896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:35.899812    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:29:35.899896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:35.899878    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:29:36.783233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.783203    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:36.866949 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.866917    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-openshift-ca\") pod \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") "
Oct 30 22:29:36.867094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.867048    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9bf3be-be04-4b49-b8fe-ba65430e9925-kube-api-access-6whdm\" (UniqueName: \"kubernetes.io/projected/4c9bf3be-be04-4b49-b8fe-ba65430e9925-kube-api-access-6whdm\") pod \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") "
Oct 30 22:29:36.867094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.867075    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-xks-ca\") pod \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\" (UID: \"4c9bf3be-be04-4b49-b8fe-ba65430e9925\") "
Oct 30 22:29:36.867250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.867229    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-openshift-ca" pod "4c9bf3be-be04-4b49-b8fe-ba65430e9925" (UID: "4c9bf3be-be04-4b49-b8fe-ba65430e9925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:36.867527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.867502    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:36.869632 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.869611    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9bf3be-be04-4b49-b8fe-ba65430e9925-kube-api-access-6whdm" pod "4c9bf3be-be04-4b49-b8fe-ba65430e9925" (UID: "4c9bf3be-be04-4b49-b8fe-ba65430e9925"). InnerVolumeSpecName "kube-api-access-6whdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:36.869770 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.869630    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-xks-ca" pod "4c9bf3be-be04-4b49-b8fe-ba65430e9925" (UID: "4c9bf3be-be04-4b49-b8fe-ba65430e9925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:36.905394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.905362    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:36.905819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.905797    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:36.906470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.906450    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:29:36.977661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.975564    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whdm\" (UniqueName: \"kubernetes.io/projected/4c9bf3be-be04-4b49-b8fe-ba65430e9925-kube-api-access-6whdm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:36.977661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:36.975604    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c9bf3be-be04-4b49-b8fe-ba65430e9925-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:37.639752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:37.639718    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"
Oct 30 22:29:37.639931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:37.639830    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h" event={"ID":"4c9bf3be-be04-4b49-b8fe-ba65430e9925","Type":"ContainerDied","Data":"332a4bc663d029852ebecefd7cda11d36820aec05e5b7ccda61a7804f3acba04"}
Oct 30 22:29:37.640032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:37.639975    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="332a4bc663d029852ebecefd7cda11d36820aec05e5b7ccda61a7804f3acba04"
Oct 30 22:29:39.123955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:39.123917    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:40.339080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.339046    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"]
Oct 30 22:29:40.339507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.339492    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4c9bf3be-be04-4b49-b8fe-ba65430e9925" containerName="revoke-keys"
Oct 30 22:29:40.339561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.339508    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9bf3be-be04-4b49-b8fe-ba65430e9925" containerName="revoke-keys"
Oct 30 22:29:40.339601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.339576    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9bf3be-be04-4b49-b8fe-ba65430e9925" containerName="revoke-keys"
Oct 30 22:29:40.343054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.343038    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.398130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.398100    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"]
Oct 30 22:29:40.445411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.445377    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:40.445411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.445379    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:40.496340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.496299    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:29:40.496340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.496312    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:40.517189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.517150    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vknv\" (UniqueName: \"kubernetes.io/projected/88414f5d-2368-4704-b519-8d5c46238481-kube-api-access-4vknv\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.517189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.517194    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88414f5d-2368-4704-b519-8d5c46238481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.517370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.517269    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88414f5d-2368-4704-b519-8d5c46238481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.618134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.618033    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vknv\" (UniqueName: \"kubernetes.io/projected/88414f5d-2368-4704-b519-8d5c46238481-kube-api-access-4vknv\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.618134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.618090    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88414f5d-2368-4704-b519-8d5c46238481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.618134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.618126    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88414f5d-2368-4704-b519-8d5c46238481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.618922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.618901    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88414f5d-2368-4704-b519-8d5c46238481-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.620496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.620476    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88414f5d-2368-4704-b519-8d5c46238481-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.681152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.681122    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vknv\" (UniqueName: \"kubernetes.io/projected/88414f5d-2368-4704-b519-8d5c46238481-kube-api-access-4vknv\") pod \"maas-api-revoke-sub-e29ba184-m8pls\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:40.955809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:40.955773    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:41.031572 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.031538    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:29:41.055761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.055733    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"]
Oct 30 22:29:41.060335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.060313    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.136188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.136153    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"]
Oct 30 22:29:41.224259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.224185    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.224259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.224229    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7g7\" (UniqueName: \"kubernetes.io/projected/00b21a54-db8e-452e-a44e-2d1788e48b9a-kube-api-access-7k7g7\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.224259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.224248    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.325729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.325695    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7g7\" (UniqueName: \"kubernetes.io/projected/00b21a54-db8e-452e-a44e-2d1788e48b9a-kube-api-access-7k7g7\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.325894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.325733    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.325979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.325962    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.326519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.326479    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.326687 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:41.326663    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88414f5d_2368_4704_b519_8d5c46238481.slice/crio-ffde4c12aac9482caea9ae4e8f3e8e3c62455b073493f16f4295d40ee70ccf6d WatchSource:0}: Error finding container ffde4c12aac9482caea9ae4e8f3e8e3c62455b073493f16f4295d40ee70ccf6d: Status 404 returned error can't find the container with id ffde4c12aac9482caea9ae4e8f3e8e3c62455b073493f16f4295d40ee70ccf6d
Oct 30 22:29:41.328721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.328673    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.329219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.329199    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"]
Oct 30 22:29:41.388041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.388011    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7g7\" (UniqueName: \"kubernetes.io/projected/00b21a54-db8e-452e-a44e-2d1788e48b9a-kube-api-access-7k7g7\") pod \"maas-api-revoke-sub-1024e4c7-8qm7p\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:41.664814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.664778    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls" event={"ID":"88414f5d-2368-4704-b519-8d5c46238481","Type":"ContainerStarted","Data":"ed230ed7765e947cd935210314bd68fba8773c53dd9d0435f2b2ca9c9436e82f"}
Oct 30 22:29:41.664990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.664839    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls" event={"ID":"88414f5d-2368-4704-b519-8d5c46238481","Type":"ContainerStarted","Data":"ffde4c12aac9482caea9ae4e8f3e8e3c62455b073493f16f4295d40ee70ccf6d"}
Oct 30 22:29:41.675189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:41.675165    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:42.056319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:42.054707    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls" podStartSLOduration=2.054686591 podStartE2EDuration="2.054686591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:41.813878997 +0000 UTC m=+2558.148455776" watchObservedRunningTime="2026-10-30 22:29:42.054686591 +0000 UTC m=+2558.389263361"
Oct 30 22:29:42.058873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:42.058838    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"]
Oct 30 22:29:42.698658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:42.698125    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p" event={"ID":"00b21a54-db8e-452e-a44e-2d1788e48b9a","Type":"ContainerStarted","Data":"469e0248a81c7d7906230bb30c1025bc0986bd17d17765ac86faea15d9d2c2fb"}
Oct 30 22:29:42.698658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:42.698193    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p" event={"ID":"00b21a54-db8e-452e-a44e-2d1788e48b9a","Type":"ContainerStarted","Data":"647e241b251af62f8d2450f7122cf9dac9fccf299f8ab9a92fdbf236aaa8e3a3"}
Oct 30 22:29:42.700527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:42.700494    5503 generic.go:356] "Generic (PLEG): container finished" podID="88414f5d-2368-4704-b519-8d5c46238481" containerID="ed230ed7765e947cd935210314bd68fba8773c53dd9d0435f2b2ca9c9436e82f" exitCode=0
Oct 30 22:29:42.700684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:42.700652    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls" event={"ID":"88414f5d-2368-4704-b519-8d5c46238481","Type":"ContainerDied","Data":"ed230ed7765e947cd935210314bd68fba8773c53dd9d0435f2b2ca9c9436e82f"}
Oct 30 22:29:42.830661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:42.829769    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p" podStartSLOduration=1.8297501760000001 podStartE2EDuration="1.829750176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:42.829020751 +0000 UTC m=+2559.163597521" watchObservedRunningTime="2026-10-30 22:29:42.829750176 +0000 UTC m=+2559.164326946"
Oct 30 22:29:43.708330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.708294    5503 generic.go:356] "Generic (PLEG): container finished" podID="00b21a54-db8e-452e-a44e-2d1788e48b9a" containerID="469e0248a81c7d7906230bb30c1025bc0986bd17d17765ac86faea15d9d2c2fb" exitCode=0
Oct 30 22:29:43.708763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.708382    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p" event={"ID":"00b21a54-db8e-452e-a44e-2d1788e48b9a","Type":"ContainerDied","Data":"469e0248a81c7d7906230bb30c1025bc0986bd17d17765ac86faea15d9d2c2fb"}
Oct 30 22:29:43.814684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.814648    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:43.880088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.880054    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:43.911954 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.911921    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88414f5d-2368-4704-b519-8d5c46238481-kube-api-access-4vknv\" (UniqueName: \"kubernetes.io/projected/88414f5d-2368-4704-b519-8d5c46238481-kube-api-access-4vknv\") pod \"88414f5d-2368-4704-b519-8d5c46238481\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") "
Oct 30 22:29:43.912160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.911976    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88414f5d-2368-4704-b519-8d5c46238481-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88414f5d-2368-4704-b519-8d5c46238481-maas-api-openshift-ca\") pod \"88414f5d-2368-4704-b519-8d5c46238481\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") "
Oct 30 22:29:43.912160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.912016    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88414f5d-2368-4704-b519-8d5c46238481-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88414f5d-2368-4704-b519-8d5c46238481-maas-api-xks-ca\") pod \"88414f5d-2368-4704-b519-8d5c46238481\" (UID: \"88414f5d-2368-4704-b519-8d5c46238481\") "
Oct 30 22:29:43.912434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.912411    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88414f5d-2368-4704-b519-8d5c46238481-maas-api-openshift-ca" pod "88414f5d-2368-4704-b519-8d5c46238481" (UID: "88414f5d-2368-4704-b519-8d5c46238481"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:43.912693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.912674    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88414f5d-2368-4704-b519-8d5c46238481-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:43.914159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.914140    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88414f5d-2368-4704-b519-8d5c46238481-maas-api-xks-ca" pod "88414f5d-2368-4704-b519-8d5c46238481" (UID: "88414f5d-2368-4704-b519-8d5c46238481"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:43.914274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:43.914254    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88414f5d-2368-4704-b519-8d5c46238481-kube-api-access-4vknv" pod "88414f5d-2368-4704-b519-8d5c46238481" (UID: "88414f5d-2368-4704-b519-8d5c46238481"). InnerVolumeSpecName "kube-api-access-4vknv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:44.013662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.013578    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vknv\" (UniqueName: \"kubernetes.io/projected/88414f5d-2368-4704-b519-8d5c46238481-kube-api-access-4vknv\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:44.013662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.013604    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88414f5d-2368-4704-b519-8d5c46238481-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:44.547897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.547863    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:44.715819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.715782    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"
Oct 30 22:29:44.715819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.715799    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls" event={"ID":"88414f5d-2368-4704-b519-8d5c46238481","Type":"ContainerDied","Data":"ffde4c12aac9482caea9ae4e8f3e8e3c62455b073493f16f4295d40ee70ccf6d"}
Oct 30 22:29:44.716359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.715867    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffde4c12aac9482caea9ae4e8f3e8e3c62455b073493f16f4295d40ee70ccf6d"
Oct 30 22:29:44.858849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.858815    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:44.922820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.922786    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-xks-ca\") pod \"00b21a54-db8e-452e-a44e-2d1788e48b9a\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") "
Oct 30 22:29:44.922820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.922824    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00b21a54-db8e-452e-a44e-2d1788e48b9a-kube-api-access-7k7g7\" (UniqueName: \"kubernetes.io/projected/00b21a54-db8e-452e-a44e-2d1788e48b9a-kube-api-access-7k7g7\") pod \"00b21a54-db8e-452e-a44e-2d1788e48b9a\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") "
Oct 30 22:29:44.923067 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.922900    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-openshift-ca\") pod \"00b21a54-db8e-452e-a44e-2d1788e48b9a\" (UID: \"00b21a54-db8e-452e-a44e-2d1788e48b9a\") "
Oct 30 22:29:44.923287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.923265    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-openshift-ca" pod "00b21a54-db8e-452e-a44e-2d1788e48b9a" (UID: "00b21a54-db8e-452e-a44e-2d1788e48b9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:44.925114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.925081    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-xks-ca" pod "00b21a54-db8e-452e-a44e-2d1788e48b9a" (UID: "00b21a54-db8e-452e-a44e-2d1788e48b9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:44.925221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:44.925165    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b21a54-db8e-452e-a44e-2d1788e48b9a-kube-api-access-7k7g7" pod "00b21a54-db8e-452e-a44e-2d1788e48b9a" (UID: "00b21a54-db8e-452e-a44e-2d1788e48b9a"). InnerVolumeSpecName "kube-api-access-7k7g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:45.024005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:45.023976    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.024005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:45.024001    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7g7\" (UniqueName: \"kubernetes.io/projected/00b21a54-db8e-452e-a44e-2d1788e48b9a-kube-api-access-7k7g7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.024005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:45.024014    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00b21a54-db8e-452e-a44e-2d1788e48b9a-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:45.723553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:45.723513    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p" event={"ID":"00b21a54-db8e-452e-a44e-2d1788e48b9a","Type":"ContainerDied","Data":"647e241b251af62f8d2450f7122cf9dac9fccf299f8ab9a92fdbf236aaa8e3a3"}
Oct 30 22:29:45.723946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:45.723601    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647e241b251af62f8d2450f7122cf9dac9fccf299f8ab9a92fdbf236aaa8e3a3"
Oct 30 22:29:45.723946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:45.723524    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"
Oct 30 22:29:47.636178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.636140    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"]
Oct 30 22:29:47.636584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.636552    5503 cpu_manager.go:414] "containerMap: removing container" podUID="88414f5d-2368-4704-b519-8d5c46238481" containerName="revoke-keys"
Oct 30 22:29:47.636584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.636563    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88414f5d-2368-4704-b519-8d5c46238481" containerName="revoke-keys"
Oct 30 22:29:47.636584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.636576    5503 cpu_manager.go:414] "containerMap: removing container" podUID="00b21a54-db8e-452e-a44e-2d1788e48b9a" containerName="revoke-keys"
Oct 30 22:29:47.636584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.636581    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00b21a54-db8e-452e-a44e-2d1788e48b9a" containerName="revoke-keys"
Oct 30 22:29:47.636750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.636693    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="88414f5d-2368-4704-b519-8d5c46238481" containerName="revoke-keys"
Oct 30 22:29:47.636750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.636702    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="00b21a54-db8e-452e-a44e-2d1788e48b9a" containerName="revoke-keys"
Oct 30 22:29:47.640080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.640064    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.751900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.751868    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zks\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-kube-api-access-t9zks\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.751900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.751901    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.752111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.751922    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.752111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.751940    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.752111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.752064    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.752111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.752105    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.752243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.752165    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.752243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.752184    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.752307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.752284    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.840111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.840080    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"]
Oct 30 22:29:47.853533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853504    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zks\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-kube-api-access-t9zks\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.853533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853536    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.853764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853568    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.853764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853703    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.853892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853810    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.853892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853874    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853952    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.853987    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.854071    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.854125    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.854149    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-data\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.854338    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-certs\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.854511    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-credential-socket\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.854863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.854835    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istiod-ca-cert\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.856028 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.856009    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-envoy\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.856339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.856321    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-podinfo\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.915789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.915758    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-token\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.915924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.915895    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zks\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-kube-api-access-t9zks\") pod \"e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") " pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:47.953022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:47.952999    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:48.382116 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:48.382084    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131bcd3e_372f_47ad_aca8_72a0a92d7d53.slice/crio-3d9aba922626dde5af2641529b32b527c4272c8349f4466d4f6c1ff4f5d5d74d WatchSource:0}: Error finding container 3d9aba922626dde5af2641529b32b527c4272c8349f4466d4f6c1ff4f5d5d74d: Status 404 returned error can't find the container with id 3d9aba922626dde5af2641529b32b527c4272c8349f4466d4f6c1ff4f5d5d74d
Oct 30 22:29:48.383666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.383630    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"]
Oct 30 22:29:48.384687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.384658    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:29:48.384745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.384722    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:29:48.384781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.384753    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:29:48.525349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.523832    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:48.747462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.747430    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" event={"ID":"131bcd3e-372f-47ad-aca8-72a0a92d7d53","Type":"ContainerStarted","Data":"c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a"}
Oct 30 22:29:48.747879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.747484    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" event={"ID":"131bcd3e-372f-47ad-aca8-72a0a92d7d53","Type":"ContainerStarted","Data":"3d9aba922626dde5af2641529b32b527c4272c8349f4466d4f6c1ff4f5d5d74d"}
Oct 30 22:29:48.879300 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.879248    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podStartSLOduration=1.8792321730000001 podStartE2EDuration="1.879232173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:48.870267081 +0000 UTC m=+2565.204843853" watchObservedRunningTime="2026-10-30 22:29:48.879232173 +0000 UTC m=+2565.213808942"
Oct 30 22:29:48.966735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:48.954212    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:49.955082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:49.955043    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:29:49.955568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:49.955108    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 22:29:50.954069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:50.954030    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:29:50.954296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:50.954104    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 22:29:51.953822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:51.953791    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:29:51.954226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:51.953843    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 22:29:52.953991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:52.953951    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:29:52.954493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:52.954179    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 22:29:53.440887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:53.440849    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.605131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:53.605088    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.805782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:53.805710    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:53.953738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:53.953702    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:29:53.953907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:53.953753    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 22:29:54.954050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:54.954019    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Oct 30 22:29:54.954498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:54.954081    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Oct 30 22:29:54.991496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:54.991465    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:55.960019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:55.959989    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:55.960542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:55.960516    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:55.961348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:55.961325    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:29:56.096914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.096878    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:56.104022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.103991    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"]
Oct 30 22:29:56.108927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.108903    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.163575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.163536    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"]
Oct 30 22:29:56.182409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.182381    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4k4\" (UniqueName: \"kubernetes.io/projected/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-kube-api-access-gk4k4\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.182687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.182662    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.182781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.182754    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.191459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.191432    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:29:56.191591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.191482    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:29:56.195822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.195791    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:29:56.195953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.195901    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:29:56.292483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.292399    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.293543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.293520    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.293831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.293808    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4k4\" (UniqueName: \"kubernetes.io/projected/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-kube-api-access-gk4k4\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.294409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.293462    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.298278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.298253    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.356153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.356125    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4k4\" (UniqueName: \"kubernetes.io/projected/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-kube-api-access-gk4k4\") pod \"maas-api-revoke-sub-8e0a62a8-mz72z\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.401062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.401021    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"]
Oct 30 22:29:56.434750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.434717    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:56.459982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.455956    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.503657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.501327    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"]
Oct 30 22:29:56.606571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.606298    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.606571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.606342    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl89x\" (UniqueName: \"kubernetes.io/projected/35726435-9e69-4cf9-87ff-c0c4cb7c6962-kube-api-access-vl89x\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.606571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.606390    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.705335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.702118    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:56.707933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.707894    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.707933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.707942    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl89x\" (UniqueName: \"kubernetes.io/projected/35726435-9e69-4cf9-87ff-c0c4cb7c6962-kube-api-access-vl89x\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.708178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.707992    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.709040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.708926    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.712487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.712463    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.771737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.771697    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl89x\" (UniqueName: \"kubernetes.io/projected/35726435-9e69-4cf9-87ff-c0c4cb7c6962-kube-api-access-vl89x\") pod \"maas-api-revoke-sub-392eea77-vljvv\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.829045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.828976    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:29:56.917240 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:56.917211    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab2066c_8fb3_40a5_81eb_2fc9f27ca4d3.slice/crio-313e0715d14dcc2985be9f739834cf3e3b72a3166115bb3338768ef15a06287e WatchSource:0}: Error finding container 313e0715d14dcc2985be9f739834cf3e3b72a3166115bb3338768ef15a06287e: Status 404 returned error can't find the container with id 313e0715d14dcc2985be9f739834cf3e3b72a3166115bb3338768ef15a06287e
Oct 30 22:29:56.920497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:56.920471    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"]
Oct 30 22:29:57.203738 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:29:57.203699    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35726435_9e69_4cf9_87ff_c0c4cb7c6962.slice/crio-b2874eba26554c2dc1b9313d0daddc7c714e7c25b54ce535c5cec33aef0e6a2a WatchSource:0}: Error finding container b2874eba26554c2dc1b9313d0daddc7c714e7c25b54ce535c5cec33aef0e6a2a: Status 404 returned error can't find the container with id b2874eba26554c2dc1b9313d0daddc7c714e7c25b54ce535c5cec33aef0e6a2a
Oct 30 22:29:57.208877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:57.208847    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"]
Oct 30 22:29:57.824623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:57.824590    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv" event={"ID":"35726435-9e69-4cf9-87ff-c0c4cb7c6962","Type":"ContainerStarted","Data":"25da2987857f700f7e1c83d1512ce034e98c291459df5d3ff00498316e799168"}
Oct 30 22:29:57.824814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:57.824677    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv" event={"ID":"35726435-9e69-4cf9-87ff-c0c4cb7c6962","Type":"ContainerStarted","Data":"b2874eba26554c2dc1b9313d0daddc7c714e7c25b54ce535c5cec33aef0e6a2a"}
Oct 30 22:29:57.826240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:57.826216    5503 generic.go:356] "Generic (PLEG): container finished" podID="0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" containerID="3d7c6a7b4ed540db49034d64b6acf91fbc75dfbfd50cc24b772142cb86b557af" exitCode=0
Oct 30 22:29:57.826377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:57.826268    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z" event={"ID":"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3","Type":"ContainerDied","Data":"3d7c6a7b4ed540db49034d64b6acf91fbc75dfbfd50cc24b772142cb86b557af"}
Oct 30 22:29:57.826377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:57.826304    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z" event={"ID":"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3","Type":"ContainerStarted","Data":"313e0715d14dcc2985be9f739834cf3e3b72a3166115bb3338768ef15a06287e"}
Oct 30 22:29:57.945809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:57.945705    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv" podStartSLOduration=1.945691128 podStartE2EDuration="1.945691128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:57.941960775 +0000 UTC m=+2574.276537544" watchObservedRunningTime="2026-10-30 22:29:57.945691128 +0000 UTC m=+2574.280267896"
Oct 30 22:29:58.098178 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:29:58.098147    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:29:58.292346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:58.292307    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:58.834058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:58.834021    5503 generic.go:356] "Generic (PLEG): container finished" podID="35726435-9e69-4cf9-87ff-c0c4cb7c6962" containerID="25da2987857f700f7e1c83d1512ce034e98c291459df5d3ff00498316e799168" exitCode=0
Oct 30 22:29:58.834262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:58.834109    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv" event={"ID":"35726435-9e69-4cf9-87ff-c0c4cb7c6962","Type":"ContainerDied","Data":"25da2987857f700f7e1c83d1512ce034e98c291459df5d3ff00498316e799168"}
Oct 30 22:29:58.974137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:58.974109    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:59.143629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.143553    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-kube-api-access-gk4k4\" (UniqueName: \"kubernetes.io/projected/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-kube-api-access-gk4k4\") pod \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") "
Oct 30 22:29:59.143629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.143620    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-xks-ca\") pod \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") "
Oct 30 22:29:59.143926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.143739    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-openshift-ca\") pod \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\" (UID: \"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3\") "
Oct 30 22:29:59.144113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.144092    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-openshift-ca" pod "0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" (UID: "0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:59.144208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.144194    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.145842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.145813    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-xks-ca" pod "0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" (UID: "0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:59.145948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.145837    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-kube-api-access-gk4k4" pod "0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" (UID: "0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3"). InnerVolumeSpecName "kube-api-access-gk4k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:59.245289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.245250    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4k4\" (UniqueName: \"kubernetes.io/projected/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-kube-api-access-gk4k4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.245289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.245292    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.319668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.319621    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:59.477422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.477379    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:29:59.842073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.842052    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"
Oct 30 22:29:59.842221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.842084    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z" event={"ID":"0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3","Type":"ContainerDied","Data":"313e0715d14dcc2985be9f739834cf3e3b72a3166115bb3338768ef15a06287e"}
Oct 30 22:29:59.842221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.842127    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="313e0715d14dcc2985be9f739834cf3e3b72a3166115bb3338768ef15a06287e"
Oct 30 22:29:59.981712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:29:59.981685    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:30:00.153180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.153144    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35726435-9e69-4cf9-87ff-c0c4cb7c6962-kube-api-access-vl89x\" (UniqueName: \"kubernetes.io/projected/35726435-9e69-4cf9-87ff-c0c4cb7c6962-kube-api-access-vl89x\") pod \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") "
Oct 30 22:30:00.153531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.153508    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-openshift-ca\") pod \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") "
Oct 30 22:30:00.153669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.153622    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-xks-ca\") pod \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\" (UID: \"35726435-9e69-4cf9-87ff-c0c4cb7c6962\") "
Oct 30 22:30:00.153973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.153949    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-openshift-ca" pod "35726435-9e69-4cf9-87ff-c0c4cb7c6962" (UID: "35726435-9e69-4cf9-87ff-c0c4cb7c6962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:00.154112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.154097    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.155411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.155380    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35726435-9e69-4cf9-87ff-c0c4cb7c6962-kube-api-access-vl89x" pod "35726435-9e69-4cf9-87ff-c0c4cb7c6962" (UID: "35726435-9e69-4cf9-87ff-c0c4cb7c6962"). InnerVolumeSpecName "kube-api-access-vl89x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:00.155744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.155722    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-xks-ca" pod "35726435-9e69-4cf9-87ff-c0c4cb7c6962" (UID: "35726435-9e69-4cf9-87ff-c0c4cb7c6962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:00.181258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.181231    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"]
Oct 30 22:30:00.181730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.181710    5503 cpu_manager.go:414] "containerMap: removing container" podUID="35726435-9e69-4cf9-87ff-c0c4cb7c6962" containerName="revoke-keys"
Oct 30 22:30:00.181730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.181729    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35726435-9e69-4cf9-87ff-c0c4cb7c6962" containerName="revoke-keys"
Oct 30 22:30:00.181922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.181795    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" containerName="revoke-keys"
Oct 30 22:30:00.181922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.181805    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" containerName="revoke-keys"
Oct 30 22:30:00.182027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.181925    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" containerName="revoke-keys"
Oct 30 22:30:00.182027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.181942    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="35726435-9e69-4cf9-87ff-c0c4cb7c6962" containerName="revoke-keys"
Oct 30 22:30:00.185167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.185149    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"]
Oct 30 22:30:00.185302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.185284    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.193698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.190745    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"]
Oct 30 22:30:00.193698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.191596    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.200939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.199234    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.213465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.213429    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"]
Oct 30 22:30:00.217620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.217600    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"]
Oct 30 22:30:00.217743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.217704    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.221493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.221472    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.221770 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.221745    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"]
Oct 30 22:30:00.225907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.225888    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.240875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.240854    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"]
Oct 30 22:30:00.255164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.255139    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35726435-9e69-4cf9-87ff-c0c4cb7c6962-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.255263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.255167    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl89x\" (UniqueName: \"kubernetes.io/projected/35726435-9e69-4cf9-87ff-c0c4cb7c6962-kube-api-access-vl89x\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.264521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.264498    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"]
Oct 30 22:30:00.288896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.287054    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"]
Oct 30 22:30:00.290528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.290492    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"]
Oct 30 22:30:00.293025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.293002    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"]
Oct 30 22:30:00.313751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.313716    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"]
Oct 30 22:30:00.356306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356279    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec7968-f189-469a-bd8b-b30ba95bb257-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.356773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356322    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2sz\" (UniqueName: \"kubernetes.io/projected/003b0aec-651e-43d0-ad83-5cebb42820b6-kube-api-access-8k2sz\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.356773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356427    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408eed43-fdb0-4490-ab1e-0d3ff281cc94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.356773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356477    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f5d72e-af25-40a9-8985-91698f555804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.356773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356537    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m57nv\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.356773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356601    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxtr\" (UniqueName: \"kubernetes.io/projected/33f5d72e-af25-40a9-8985-91698f555804-kube-api-access-vrxtr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.356773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356705    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1369e2-c33d-49d6-8c4e-76da990030e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.357020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356787    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003b0aec-651e-43d0-ad83-5cebb42820b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.357020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356806    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vlgd\" (UniqueName: \"kubernetes.io/projected/68ec7968-f189-469a-bd8b-b30ba95bb257-kube-api-access-2vlgd\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.357020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356824    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwbr\" (UniqueName: \"kubernetes.io/projected/408eed43-fdb0-4490-ab1e-0d3ff281cc94-kube-api-access-drwbr\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.357020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356857    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbbd\" (UniqueName: \"kubernetes.io/projected/4e1369e2-c33d-49d6-8c4e-76da990030e2-kube-api-access-ljbbd\") pod \"maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.357020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.356946    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmqx\" (UniqueName: \"kubernetes.io/projected/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-kube-api-access-8pmqx\") pod \"maas-api-key-cleanup-29889990-m57nv\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.457826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.457742    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxtr\" (UniqueName: \"kubernetes.io/projected/33f5d72e-af25-40a9-8985-91698f555804-kube-api-access-vrxtr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.457826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.457820    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1369e2-c33d-49d6-8c4e-76da990030e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.458048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.457883    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003b0aec-651e-43d0-ad83-5cebb42820b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.458048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.457906    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vlgd\" (UniqueName: \"kubernetes.io/projected/68ec7968-f189-469a-bd8b-b30ba95bb257-kube-api-access-2vlgd\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.458048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.457930    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drwbr\" (UniqueName: \"kubernetes.io/projected/408eed43-fdb0-4490-ab1e-0d3ff281cc94-kube-api-access-drwbr\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.458048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.457970    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbbd\" (UniqueName: \"kubernetes.io/projected/4e1369e2-c33d-49d6-8c4e-76da990030e2-kube-api-access-ljbbd\") pod \"maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.458048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458019    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmqx\" (UniqueName: \"kubernetes.io/projected/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-kube-api-access-8pmqx\") pod \"maas-api-key-cleanup-29889990-m57nv\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.458371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458066    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec7968-f189-469a-bd8b-b30ba95bb257-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.458371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458113    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2sz\" (UniqueName: \"kubernetes.io/projected/003b0aec-651e-43d0-ad83-5cebb42820b6-kube-api-access-8k2sz\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.458371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458169    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408eed43-fdb0-4490-ab1e-0d3ff281cc94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.458371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458200    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f5d72e-af25-40a9-8985-91698f555804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.458371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458235    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m57nv\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.458790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458763    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003b0aec-651e-43d0-ad83-5cebb42820b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.458890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458765    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1369e2-c33d-49d6-8c4e-76da990030e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.458890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.458817    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec7968-f189-469a-bd8b-b30ba95bb257-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.459084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.459061    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408eed43-fdb0-4490-ab1e-0d3ff281cc94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.459153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.459096    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f5d72e-af25-40a9-8985-91698f555804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.459153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.459103    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m57nv\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.527380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.527351    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbbd\" (UniqueName: \"kubernetes.io/projected/4e1369e2-c33d-49d6-8c4e-76da990030e2-kube-api-access-ljbbd\") pod \"maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.527380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.527366    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxtr\" (UniqueName: \"kubernetes.io/projected/33f5d72e-af25-40a9-8985-91698f555804-kube-api-access-vrxtr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.527581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.527539    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vlgd\" (UniqueName: \"kubernetes.io/projected/68ec7968-f189-469a-bd8b-b30ba95bb257-kube-api-access-2vlgd\") pod \"maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.527664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.527597    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2sz\" (UniqueName: \"kubernetes.io/projected/003b0aec-651e-43d0-ad83-5cebb42820b6-kube-api-access-8k2sz\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.527944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.527924    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmqx\" (UniqueName: \"kubernetes.io/projected/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-kube-api-access-8pmqx\") pod \"maas-api-key-cleanup-29889990-m57nv\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.528007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.527991    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwbr\" (UniqueName: \"kubernetes.io/projected/408eed43-fdb0-4490-ab1e-0d3ff281cc94-kube-api-access-drwbr\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.534321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.534307    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:00.545938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.545906    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:00.557144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.557126    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:00.806660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.806538    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:00.815841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.815819    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:00.823895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.823877    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:00.852541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.852512    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"
Oct 30 22:30:00.852706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.852545    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv" event={"ID":"35726435-9e69-4cf9-87ff-c0c4cb7c6962","Type":"ContainerDied","Data":"b2874eba26554c2dc1b9313d0daddc7c714e7c25b54ce535c5cec33aef0e6a2a"}
Oct 30 22:30:00.852706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.852604    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2874eba26554c2dc1b9313d0daddc7c714e7c25b54ce535c5cec33aef0e6a2a"
Oct 30 22:30:00.882183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.877595    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:00.914862 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:00.914816    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408eed43_fdb0_4490_ab1e_0d3ff281cc94.slice/crio-db9fde917709717590dc7e331e11861cae1a3a7a5d59d264b502e71281593644 WatchSource:0}: Error finding container db9fde917709717590dc7e331e11861cae1a3a7a5d59d264b502e71281593644: Status 404 returned error can't find the container with id db9fde917709717590dc7e331e11861cae1a3a7a5d59d264b502e71281593644
Oct 30 22:30:00.917330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.916746    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"]
Oct 30 22:30:00.961928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:00.961903    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"]
Oct 30 22:30:01.050215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.050180    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:01.073063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.073037    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"]
Oct 30 22:30:01.117183 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:01.117151    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1369e2_c33d_49d6_8c4e_76da990030e2.slice/crio-27eef3909fdd42b8efb06d64eec47c31292fb5dabaa5952d135d6aee3f019354 WatchSource:0}: Error finding container 27eef3909fdd42b8efb06d64eec47c31292fb5dabaa5952d135d6aee3f019354: Status 404 returned error can't find the container with id 27eef3909fdd42b8efb06d64eec47c31292fb5dabaa5952d135d6aee3f019354
Oct 30 22:30:01.201624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.201596    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"]
Oct 30 22:30:01.204625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.204602    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:01.217263 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:01.217232    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1858e3e4_bdfe_4ba9_91d4_f79e2fad5f40.slice/crio-3943d4d8cf7fbde383ed1c76cf3d224a6a476c1cbc775dee31b9f4493b85fe1d WatchSource:0}: Error finding container 3943d4d8cf7fbde383ed1c76cf3d224a6a476c1cbc775dee31b9f4493b85fe1d: Status 404 returned error can't find the container with id 3943d4d8cf7fbde383ed1c76cf3d224a6a476c1cbc775dee31b9f4493b85fe1d
Oct 30 22:30:01.253284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.253253    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"]
Oct 30 22:30:01.256919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.256899    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"]
Oct 30 22:30:01.257249 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:01.257226    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f5d72e_af25_40a9_8985_91698f555804.slice/crio-5bd978c707d55691efbfd586c2785bfbf41d09ec682b3dfcaf288be1b72ff32a WatchSource:0}: Error finding container 5bd978c707d55691efbfd586c2785bfbf41d09ec682b3dfcaf288be1b72ff32a: Status 404 returned error can't find the container with id 5bd978c707d55691efbfd586c2785bfbf41d09ec682b3dfcaf288be1b72ff32a
Oct 30 22:30:01.365181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.365158    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:01.868576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.868534    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" event={"ID":"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40","Type":"ContainerStarted","Data":"3e4d94a33208707149ab2a5f47036a451619a86379b0f757dfbcfd4bf54d5fe0"}
Oct 30 22:30:01.868797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.868629    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" event={"ID":"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40","Type":"ContainerStarted","Data":"3943d4d8cf7fbde383ed1c76cf3d224a6a476c1cbc775dee31b9f4493b85fe1d"}
Oct 30 22:30:01.870362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.870335    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerStarted","Data":"b7a3027c467a75d97c2376647f1fe957787a2f9fc53eec01675aaff2e9142ece"}
Oct 30 22:30:01.870495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.870391    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerStarted","Data":"5bd978c707d55691efbfd586c2785bfbf41d09ec682b3dfcaf288be1b72ff32a"}
Oct 30 22:30:01.872263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.872203    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerStarted","Data":"4e579ed93ebbb59f98f8b0b0f3070a9eec6490bf54bebd99fe172f526366ec92"}
Oct 30 22:30:01.872263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.872243    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerStarted","Data":"42e6b5be6c8c40dfae10735c9ca7e429b013b7b37f5a06b536837e3fcb6f1f54"}
Oct 30 22:30:01.873927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.873899    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" event={"ID":"68ec7968-f189-469a-bd8b-b30ba95bb257","Type":"ContainerStarted","Data":"031cbc3dfe4e9995df90ffa56349fd462c158168a9a45e5f2c7f7d1eda044064"}
Oct 30 22:30:01.874023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.873948    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" event={"ID":"68ec7968-f189-469a-bd8b-b30ba95bb257","Type":"ContainerStarted","Data":"7fe48e20daf2dfbf97c18f815684802c9cd40f0080469266ee567bb478f9f435"}
Oct 30 22:30:01.875369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.875339    5503 generic.go:356] "Generic (PLEG): container finished" podID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerID="f23196521da18f5022006723edec115cc36ae617f49811731e58c42dee805ae6" exitCode=60
Oct 30 22:30:01.875446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.875411    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" event={"ID":"408eed43-fdb0-4490-ab1e-0d3ff281cc94","Type":"ContainerDied","Data":"f23196521da18f5022006723edec115cc36ae617f49811731e58c42dee805ae6"}
Oct 30 22:30:01.875501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.875456    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" event={"ID":"408eed43-fdb0-4490-ab1e-0d3ff281cc94","Type":"ContainerStarted","Data":"db9fde917709717590dc7e331e11861cae1a3a7a5d59d264b502e71281593644"}
Oct 30 22:30:01.875596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.875580    5503 scope.go:122] "RemoveContainer" containerID="f23196521da18f5022006723edec115cc36ae617f49811731e58c42dee805ae6"
Oct 30 22:30:01.877112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.877088    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" event={"ID":"4e1369e2-c33d-49d6-8c4e-76da990030e2","Type":"ContainerStarted","Data":"6fb5267210b3b19161fb563a1294ca226d3c26a6a18631e35acb3e86d4947d8b"}
Oct 30 22:30:01.877207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:01.877133    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" event={"ID":"4e1369e2-c33d-49d6-8c4e-76da990030e2","Type":"ContainerStarted","Data":"27eef3909fdd42b8efb06d64eec47c31292fb5dabaa5952d135d6aee3f019354"}
Oct 30 22:30:02.064023 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:02.063988    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1369e2_c33d_49d6_8c4e_76da990030e2.slice/crio-conmon-6fb5267210b3b19161fb563a1294ca226d3c26a6a18631e35acb3e86d4947d8b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1369e2_c33d_49d6_8c4e_76da990030e2.slice/crio-6fb5267210b3b19161fb563a1294ca226d3c26a6a18631e35acb3e86d4947d8b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:30:02.104882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.104836    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" podStartSLOduration=2.104822074 podStartE2EDuration="2.104822074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.985685506 +0000 UTC m=+2578.320262275" watchObservedRunningTime="2026-10-30 22:30:02.104822074 +0000 UTC m=+2578.439398889"
Oct 30 22:30:02.105695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.105667    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" podStartSLOduration=2.105658761 podStartE2EDuration="2.105658761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.102560536 +0000 UTC m=+2578.437137305" watchObservedRunningTime="2026-10-30 22:30:02.105658761 +0000 UTC m=+2578.440235530"
Oct 30 22:30:02.225235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.225184    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" podStartSLOduration=2.225168912 podStartE2EDuration="2.225168912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.221335432 +0000 UTC m=+2578.555912201" watchObservedRunningTime="2026-10-30 22:30:02.225168912 +0000 UTC m=+2578.559745680"
Oct 30 22:30:02.339806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.339753    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" podStartSLOduration=2.339739621 podStartE2EDuration="2.339739621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.337896342 +0000 UTC m=+2578.672473122" watchObservedRunningTime="2026-10-30 22:30:02.339739621 +0000 UTC m=+2578.674316445"
Oct 30 22:30:02.454990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.454937    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" podStartSLOduration=2.454922608 podStartE2EDuration="2.454922608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.454353088 +0000 UTC m=+2578.788929858" watchObservedRunningTime="2026-10-30 22:30:02.454922608 +0000 UTC m=+2578.789499377"
Oct 30 22:30:02.885081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.884996    5503 generic.go:356] "Generic (PLEG): container finished" podID="33f5d72e-af25-40a9-8985-91698f555804" containerID="b7a3027c467a75d97c2376647f1fe957787a2f9fc53eec01675aaff2e9142ece" exitCode=60
Oct 30 22:30:02.885233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.885082    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerDied","Data":"b7a3027c467a75d97c2376647f1fe957787a2f9fc53eec01675aaff2e9142ece"}
Oct 30 22:30:02.885386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.885368    5503 scope.go:122] "RemoveContainer" containerID="b7a3027c467a75d97c2376647f1fe957787a2f9fc53eec01675aaff2e9142ece"
Oct 30 22:30:02.887001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.886876    5503 generic.go:356] "Generic (PLEG): container finished" podID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerID="4e579ed93ebbb59f98f8b0b0f3070a9eec6490bf54bebd99fe172f526366ec92" exitCode=60
Oct 30 22:30:02.887001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.886962    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerDied","Data":"4e579ed93ebbb59f98f8b0b0f3070a9eec6490bf54bebd99fe172f526366ec92"}
Oct 30 22:30:02.887245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.887225    5503 scope.go:122] "RemoveContainer" containerID="4e579ed93ebbb59f98f8b0b0f3070a9eec6490bf54bebd99fe172f526366ec92"
Oct 30 22:30:02.888880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.888854    5503 generic.go:356] "Generic (PLEG): container finished" podID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerID="031cbc3dfe4e9995df90ffa56349fd462c158168a9a45e5f2c7f7d1eda044064" exitCode=60
Oct 30 22:30:02.889002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.888943    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" event={"ID":"68ec7968-f189-469a-bd8b-b30ba95bb257","Type":"ContainerDied","Data":"031cbc3dfe4e9995df90ffa56349fd462c158168a9a45e5f2c7f7d1eda044064"}
Oct 30 22:30:02.889222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.889205    5503 scope.go:122] "RemoveContainer" containerID="031cbc3dfe4e9995df90ffa56349fd462c158168a9a45e5f2c7f7d1eda044064"
Oct 30 22:30:02.891084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.891067    5503 generic.go:356] "Generic (PLEG): container finished" podID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerID="198c75f22e5a6fb3108fb7b5e814faf3962f937d2ce498fe23abafc324a18746" exitCode=60
Oct 30 22:30:02.891179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.891149    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" event={"ID":"408eed43-fdb0-4490-ab1e-0d3ff281cc94","Type":"ContainerDied","Data":"198c75f22e5a6fb3108fb7b5e814faf3962f937d2ce498fe23abafc324a18746"}
Oct 30 22:30:02.891257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.891200    5503 scope.go:122] "RemoveContainer" containerID="f23196521da18f5022006723edec115cc36ae617f49811731e58c42dee805ae6"
Oct 30 22:30:02.891318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.891301    5503 scope.go:122] "RemoveContainer" containerID="198c75f22e5a6fb3108fb7b5e814faf3962f937d2ce498fe23abafc324a18746"
Oct 30 22:30:02.891517 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:02.891495    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7_odh-ai-gateway-infra(408eed43-fdb0-4490-ab1e-0d3ff281cc94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94"
Oct 30 22:30:02.892716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.892697    5503 generic.go:356] "Generic (PLEG): container finished" podID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerID="6fb5267210b3b19161fb563a1294ca226d3c26a6a18631e35acb3e86d4947d8b" exitCode=60
Oct 30 22:30:02.892821 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.892755    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" event={"ID":"4e1369e2-c33d-49d6-8c4e-76da990030e2","Type":"ContainerDied","Data":"6fb5267210b3b19161fb563a1294ca226d3c26a6a18631e35acb3e86d4947d8b"}
Oct 30 22:30:02.892926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.892911    5503 scope.go:122] "RemoveContainer" containerID="6fb5267210b3b19161fb563a1294ca226d3c26a6a18631e35acb3e86d4947d8b"
Oct 30 22:30:02.894382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.894360    5503 generic.go:356] "Generic (PLEG): container finished" podID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerID="3e4d94a33208707149ab2a5f47036a451619a86379b0f757dfbcfd4bf54d5fe0" exitCode=60
Oct 30 22:30:02.894463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.894408    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" event={"ID":"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40","Type":"ContainerDied","Data":"3e4d94a33208707149ab2a5f47036a451619a86379b0f757dfbcfd4bf54d5fe0"}
Oct 30 22:30:02.894594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:02.894582    5503 scope.go:122] "RemoveContainer" containerID="3e4d94a33208707149ab2a5f47036a451619a86379b0f757dfbcfd4bf54d5fe0"
Oct 30 22:30:03.145314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.144472    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:03.903018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.902984    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" event={"ID":"68ec7968-f189-469a-bd8b-b30ba95bb257","Type":"ContainerStarted","Data":"2a6eae9feb3bc7b452370e886229d307d93e7bb5cf3a0e98dd54c83e11a2d4c5"}
Oct 30 22:30:03.905021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.904998    5503 scope.go:122] "RemoveContainer" containerID="198c75f22e5a6fb3108fb7b5e814faf3962f937d2ce498fe23abafc324a18746"
Oct 30 22:30:03.905350 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:03.905269    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7_odh-ai-gateway-infra(408eed43-fdb0-4490-ab1e-0d3ff281cc94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94"
Oct 30 22:30:03.906799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.906771    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" event={"ID":"4e1369e2-c33d-49d6-8c4e-76da990030e2","Type":"ContainerStarted","Data":"12ea40f573228e6103a59939e0afd61165b2b83cc10b58024f09ec67b32d3df9"}
Oct 30 22:30:03.908859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.908836    5503 generic.go:356] "Generic (PLEG): container finished" podID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerID="62fada5e93f172a2ca29faf4bbd6e5774d0d16e6b68fa2edb69e14740bf99194" exitCode=60
Oct 30 22:30:03.908969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.908914    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" event={"ID":"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40","Type":"ContainerDied","Data":"62fada5e93f172a2ca29faf4bbd6e5774d0d16e6b68fa2edb69e14740bf99194"}
Oct 30 22:30:03.909028 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.908971    5503 scope.go:122] "RemoveContainer" containerID="3e4d94a33208707149ab2a5f47036a451619a86379b0f757dfbcfd4bf54d5fe0"
Oct 30 22:30:03.909167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.909152    5503 scope.go:122] "RemoveContainer" containerID="62fada5e93f172a2ca29faf4bbd6e5774d0d16e6b68fa2edb69e14740bf99194"
Oct 30 22:30:03.909398 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:03.909376    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m57nv_odh-ai-gateway-infra(1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40"
Oct 30 22:30:03.911127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.911100    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerStarted","Data":"7be03fa192697573d765fafc0940430373cea5d91fdde600935777e7e3c02f60"}
Oct 30 22:30:03.913084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:03.913058    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerStarted","Data":"dacb2a33d6ded3c432c5d8d653a05a2b14bab902aaed4faf401276fc16559dda"}
Oct 30 22:30:04.147275 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.147244    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:04.336982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.336903    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:04.564558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.564524    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"]
Oct 30 22:30:04.568469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.568442    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.622293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.622219    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9717a8\""
Oct 30 22:30:04.623102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.623076    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"]
Oct 30 22:30:04.625377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.625353    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69459b23-55f8-4c8b-8638-5f52087ea46f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.625516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.625498    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5gh\" (UniqueName: \"kubernetes.io/projected/69459b23-55f8-4c8b-8638-5f52087ea46f-kube-api-access-6c5gh\") pod \"payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.625611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.625589    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9717a8-dockercfg-m2xqb\""
Oct 30 22:30:04.736655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.726831    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69459b23-55f8-4c8b-8638-5f52087ea46f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.736655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.726967    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5gh\" (UniqueName: \"kubernetes.io/projected/69459b23-55f8-4c8b-8638-5f52087ea46f-kube-api-access-6c5gh\") pod \"payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.736655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.728356    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69459b23-55f8-4c8b-8638-5f52087ea46f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.790220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.790190    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5gh\" (UniqueName: \"kubernetes.io/projected/69459b23-55f8-4c8b-8638-5f52087ea46f-kube-api-access-6c5gh\") pod \"payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.881793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.881707    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:04.923248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.923209    5503 generic.go:356] "Generic (PLEG): container finished" podID="33f5d72e-af25-40a9-8985-91698f555804" containerID="7be03fa192697573d765fafc0940430373cea5d91fdde600935777e7e3c02f60" exitCode=60
Oct 30 22:30:04.923725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.923268    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerDied","Data":"7be03fa192697573d765fafc0940430373cea5d91fdde600935777e7e3c02f60"}
Oct 30 22:30:04.923725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.923324    5503 scope.go:122] "RemoveContainer" containerID="b7a3027c467a75d97c2376647f1fe957787a2f9fc53eec01675aaff2e9142ece"
Oct 30 22:30:04.923725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.923587    5503 scope.go:122] "RemoveContainer" containerID="7be03fa192697573d765fafc0940430373cea5d91fdde600935777e7e3c02f60"
Oct 30 22:30:04.923980 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:04.923957    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h_odh-ai-gateway-infra(33f5d72e-af25-40a9-8985-91698f555804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" podUID="33f5d72e-af25-40a9-8985-91698f555804"
Oct 30 22:30:04.925991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.925569    5503 generic.go:356] "Generic (PLEG): container finished" podID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerID="dacb2a33d6ded3c432c5d8d653a05a2b14bab902aaed4faf401276fc16559dda" exitCode=60
Oct 30 22:30:04.925991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.925820    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerDied","Data":"dacb2a33d6ded3c432c5d8d653a05a2b14bab902aaed4faf401276fc16559dda"}
Oct 30 22:30:04.926249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.926218    5503 scope.go:122] "RemoveContainer" containerID="dacb2a33d6ded3c432c5d8d653a05a2b14bab902aaed4faf401276fc16559dda"
Oct 30 22:30:04.926531 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:04.926508    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk_odh-ai-gateway-infra(003b0aec-651e-43d0-ad83-5cebb42820b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6"
Oct 30 22:30:04.928024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.928003    5503 generic.go:356] "Generic (PLEG): container finished" podID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerID="2a6eae9feb3bc7b452370e886229d307d93e7bb5cf3a0e98dd54c83e11a2d4c5" exitCode=60
Oct 30 22:30:04.928129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.928052    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" event={"ID":"68ec7968-f189-469a-bd8b-b30ba95bb257","Type":"ContainerDied","Data":"2a6eae9feb3bc7b452370e886229d307d93e7bb5cf3a0e98dd54c83e11a2d4c5"}
Oct 30 22:30:04.928468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.928451    5503 scope.go:122] "RemoveContainer" containerID="2a6eae9feb3bc7b452370e886229d307d93e7bb5cf3a0e98dd54c83e11a2d4c5"
Oct 30 22:30:04.928759 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:04.928738    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d_odh-ai-gateway-infra(68ec7968-f189-469a-bd8b-b30ba95bb257)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257"
Oct 30 22:30:04.930255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.930231    5503 generic.go:356] "Generic (PLEG): container finished" podID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerID="12ea40f573228e6103a59939e0afd61165b2b83cc10b58024f09ec67b32d3df9" exitCode=60
Oct 30 22:30:04.930352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.930269    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" event={"ID":"4e1369e2-c33d-49d6-8c4e-76da990030e2","Type":"ContainerDied","Data":"12ea40f573228e6103a59939e0afd61165b2b83cc10b58024f09ec67b32d3df9"}
Oct 30 22:30:04.930533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.930512    5503 scope.go:122] "RemoveContainer" containerID="12ea40f573228e6103a59939e0afd61165b2b83cc10b58024f09ec67b32d3df9"
Oct 30 22:30:04.930771 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:04.930752    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56_odh-ai-gateway-infra(4e1369e2-c33d-49d6-8c4e-76da990030e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2"
Oct 30 22:30:04.932516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.932500    5503 scope.go:122] "RemoveContainer" containerID="62fada5e93f172a2ca29faf4bbd6e5774d0d16e6b68fa2edb69e14740bf99194"
Oct 30 22:30:04.932692 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:04.932671    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m57nv_odh-ai-gateway-infra(1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40"
Oct 30 22:30:04.941840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.941823    5503 scope.go:122] "RemoveContainer" containerID="4e579ed93ebbb59f98f8b0b0f3070a9eec6490bf54bebd99fe172f526366ec92"
Oct 30 22:30:04.967192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.967164    5503 scope.go:122] "RemoveContainer" containerID="031cbc3dfe4e9995df90ffa56349fd462c158168a9a45e5f2c7f7d1eda044064"
Oct 30 22:30:04.988132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:04.986015    5503 scope.go:122] "RemoveContainer" containerID="6fb5267210b3b19161fb563a1294ca226d3c26a6a18631e35acb3e86d4947d8b"
Oct 30 22:30:05.264808 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:05.264780    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69459b23_55f8_4c8b_8638_5f52087ea46f.slice/crio-f9e3444dcac4a108f7744aaa3dad792595273762a11dd658ed5bb8156d28df45 WatchSource:0}: Error finding container f9e3444dcac4a108f7744aaa3dad792595273762a11dd658ed5bb8156d28df45: Status 404 returned error can't find the container with id f9e3444dcac4a108f7744aaa3dad792595273762a11dd658ed5bb8156d28df45
Oct 30 22:30:05.268094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.268066    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"]
Oct 30 22:30:05.942431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.942404    5503 scope.go:122] "RemoveContainer" containerID="7be03fa192697573d765fafc0940430373cea5d91fdde600935777e7e3c02f60"
Oct 30 22:30:05.942938 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:05.942705    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h_odh-ai-gateway-infra(33f5d72e-af25-40a9-8985-91698f555804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" podUID="33f5d72e-af25-40a9-8985-91698f555804"
Oct 30 22:30:05.944024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.943992    5503 scope.go:122] "RemoveContainer" containerID="dacb2a33d6ded3c432c5d8d653a05a2b14bab902aaed4faf401276fc16559dda"
Oct 30 22:30:05.944272 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:05.944244    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk_odh-ai-gateway-infra(003b0aec-651e-43d0-ad83-5cebb42820b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6"
Oct 30 22:30:05.945487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.945463    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" event={"ID":"69459b23-55f8-4c8b-8638-5f52087ea46f","Type":"ContainerStarted","Data":"e2ee731ca1624ad53ea1501bd4ebac96cb11f0d5c8d31c0843c7e63c8b42d688"}
Oct 30 22:30:05.945607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.945509    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" event={"ID":"69459b23-55f8-4c8b-8638-5f52087ea46f","Type":"ContainerStarted","Data":"f9e3444dcac4a108f7744aaa3dad792595273762a11dd658ed5bb8156d28df45"}
Oct 30 22:30:05.945693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.945604    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:05.947442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.947425    5503 scope.go:122] "RemoveContainer" containerID="2a6eae9feb3bc7b452370e886229d307d93e7bb5cf3a0e98dd54c83e11a2d4c5"
Oct 30 22:30:05.947669 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:05.947624    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d_odh-ai-gateway-infra(68ec7968-f189-469a-bd8b-b30ba95bb257)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257"
Oct 30 22:30:05.949152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:05.949133    5503 scope.go:122] "RemoveContainer" containerID="12ea40f573228e6103a59939e0afd61165b2b83cc10b58024f09ec67b32d3df9"
Oct 30 22:30:05.949365 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:05.949345    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56_odh-ai-gateway-infra(4e1369e2-c33d-49d6-8c4e-76da990030e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2"
Oct 30 22:30:06.529137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:06.529082    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" podStartSLOduration=2.529068407 podStartE2EDuration="2.529068407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:06.527681243 +0000 UTC m=+2582.862258009" watchObservedRunningTime="2026-10-30 22:30:06.529068407 +0000 UTC m=+2582.863645214"
Oct 30 22:30:07.188655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.188609    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"]
Oct 30 22:30:07.192416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.192396    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.248501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.248462    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"]
Oct 30 22:30:07.251900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.251878    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:07.261421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.261396    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.261531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.261429    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.261575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.261544    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgwm\" (UniqueName: \"kubernetes.io/projected/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-kube-api-access-nwgwm\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.362361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.362324    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.362361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.362365    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.362607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.362421    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgwm\" (UniqueName: \"kubernetes.io/projected/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-kube-api-access-nwgwm\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.362978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.362959    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.364780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.364762    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.425364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.425325    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgwm\" (UniqueName: \"kubernetes.io/projected/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-kube-api-access-nwgwm\") pod \"maas-api-revoke-sub-7b0fa2f2-7t799\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.506933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.506840    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:07.903838 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:07.903794    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd82aa66_e922_478f_bdcb_a75b3bb5ef86.slice/crio-49e84e7fce4fb9e42d4fd529bc96526156e4d71f5f5da4c04a75c480e6438a6f WatchSource:0}: Error finding container 49e84e7fce4fb9e42d4fd529bc96526156e4d71f5f5da4c04a75c480e6438a6f: Status 404 returned error can't find the container with id 49e84e7fce4fb9e42d4fd529bc96526156e4d71f5f5da4c04a75c480e6438a6f
Oct 30 22:30:07.907509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.907481    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"]
Oct 30 22:30:07.967148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:07.967114    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799" event={"ID":"fd82aa66-e922-478f-bdcb-a75b3bb5ef86","Type":"ContainerStarted","Data":"49e84e7fce4fb9e42d4fd529bc96526156e4d71f5f5da4c04a75c480e6438a6f"}
Oct 30 22:30:08.089049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:08.088994    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:08.258786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:08.258736    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:08.958662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:08.950181    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:09.132021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:09.131979    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799" event={"ID":"fd82aa66-e922-478f-bdcb-a75b3bb5ef86","Type":"ContainerStarted","Data":"775d1659a17c7371617883fcc3143909b1a6e4c343a1e9ecbf92dffeb3169c9d"}
Oct 30 22:30:10.156654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:10.148380    5503 generic.go:356] "Generic (PLEG): container finished" podID="fd82aa66-e922-478f-bdcb-a75b3bb5ef86" containerID="775d1659a17c7371617883fcc3143909b1a6e4c343a1e9ecbf92dffeb3169c9d" exitCode=0
Oct 30 22:30:10.156654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:10.148675    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799" event={"ID":"fd82aa66-e922-478f-bdcb-a75b3bb5ef86","Type":"ContainerDied","Data":"775d1659a17c7371617883fcc3143909b1a6e4c343a1e9ecbf92dffeb3169c9d"}
Oct 30 22:30:11.315095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.315070    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:11.337941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.337902    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-kube-api-access-nwgwm\" (UniqueName: \"kubernetes.io/projected/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-kube-api-access-nwgwm\") pod \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") "
Oct 30 22:30:11.338103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.337982    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-xks-ca\") pod \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") "
Oct 30 22:30:11.338103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.338065    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-openshift-ca\") pod \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\" (UID: \"fd82aa66-e922-478f-bdcb-a75b3bb5ef86\") "
Oct 30 22:30:11.338729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.338695    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-openshift-ca" pod "fd82aa66-e922-478f-bdcb-a75b3bb5ef86" (UID: "fd82aa66-e922-478f-bdcb-a75b3bb5ef86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:11.340208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.340182    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-kube-api-access-nwgwm" pod "fd82aa66-e922-478f-bdcb-a75b3bb5ef86" (UID: "fd82aa66-e922-478f-bdcb-a75b3bb5ef86"). InnerVolumeSpecName "kube-api-access-nwgwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:11.340208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.340198    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-xks-ca" pod "fd82aa66-e922-478f-bdcb-a75b3bb5ef86" (UID: "fd82aa66-e922-478f-bdcb-a75b3bb5ef86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:11.439388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.439313    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwgwm\" (UniqueName: \"kubernetes.io/projected/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-kube-api-access-nwgwm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.439388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.439345    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.439388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.439354    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd82aa66-e922-478f-bdcb-a75b3bb5ef86-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:11.717149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:11.716991    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:12.052623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:12.052532    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"]
Oct 30 22:30:12.055929 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:12.055900    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb5a856-l9lsn"]
Oct 30 22:30:12.163169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:12.163133    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799" event={"ID":"fd82aa66-e922-478f-bdcb-a75b3bb5ef86","Type":"ContainerDied","Data":"49e84e7fce4fb9e42d4fd529bc96526156e4d71f5f5da4c04a75c480e6438a6f"}
Oct 30 22:30:12.163169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:12.163155    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"
Oct 30 22:30:12.163403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:12.163184    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e84e7fce4fb9e42d4fd529bc96526156e4d71f5f5da4c04a75c480e6438a6f"
Oct 30 22:30:12.982682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:12.982627    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:13.786111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:13.786081    5503 scope.go:122] "RemoveContainer" containerID="198c75f22e5a6fb3108fb7b5e814faf3962f937d2ce498fe23abafc324a18746"
Oct 30 22:30:13.790248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:13.790227    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9f5047-0b99-4ddc-9482-3cfdcb2b26df" path="/var/lib/kubelet/pods/dd9f5047-0b99-4ddc-9482-3cfdcb2b26df/volumes"
Oct 30 22:30:14.177813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:14.177772    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" event={"ID":"408eed43-fdb0-4490-ab1e-0d3ff281cc94","Type":"ContainerStarted","Data":"8cda84ed74e0926cb4bdcdf92f8c7cdcc7feb65b0833482442c3ef577d17040b"}
Oct 30 22:30:14.299683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:14.299615    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" podStartSLOduration=14.299601972 podStartE2EDuration="14.299601972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:14.298610175 +0000 UTC m=+2590.633186944" watchObservedRunningTime="2026-10-30 22:30:14.299601972 +0000 UTC m=+2590.634178741"
Oct 30 22:30:14.540758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:14.540719    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:14.724625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:14.724529    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:14.898698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:14.898599    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:15.186541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:15.186508    5503 generic.go:356] "Generic (PLEG): container finished" podID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerID="8cda84ed74e0926cb4bdcdf92f8c7cdcc7feb65b0833482442c3ef577d17040b" exitCode=60
Oct 30 22:30:15.186999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:15.186576    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" event={"ID":"408eed43-fdb0-4490-ab1e-0d3ff281cc94","Type":"ContainerDied","Data":"8cda84ed74e0926cb4bdcdf92f8c7cdcc7feb65b0833482442c3ef577d17040b"}
Oct 30 22:30:15.186999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:15.186655    5503 scope.go:122] "RemoveContainer" containerID="198c75f22e5a6fb3108fb7b5e814faf3962f937d2ce498fe23abafc324a18746"
Oct 30 22:30:15.186999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:15.186983    5503 scope.go:122] "RemoveContainer" containerID="8cda84ed74e0926cb4bdcdf92f8c7cdcc7feb65b0833482442c3ef577d17040b"
Oct 30 22:30:15.187250 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:15.187221    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7_odh-ai-gateway-infra(408eed43-fdb0-4490-ab1e-0d3ff281cc94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94"
Oct 30 22:30:15.324607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:15.324541    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"]
Oct 30 22:30:16.339983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.339960    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:16.396053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.396026    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408eed43-fdb0-4490-ab1e-0d3ff281cc94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408eed43-fdb0-4490-ab1e-0d3ff281cc94-maas-api-service-ca\") pod \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") "
Oct 30 22:30:16.396218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.396116    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408eed43-fdb0-4490-ab1e-0d3ff281cc94-kube-api-access-drwbr\" (UniqueName: \"kubernetes.io/projected/408eed43-fdb0-4490-ab1e-0d3ff281cc94-kube-api-access-drwbr\") pod \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\" (UID: \"408eed43-fdb0-4490-ab1e-0d3ff281cc94\") "
Oct 30 22:30:16.396366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.396344    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408eed43-fdb0-4490-ab1e-0d3ff281cc94-maas-api-service-ca" pod "408eed43-fdb0-4490-ab1e-0d3ff281cc94" (UID: "408eed43-fdb0-4490-ab1e-0d3ff281cc94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:16.398236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.398214    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408eed43-fdb0-4490-ab1e-0d3ff281cc94-kube-api-access-drwbr" pod "408eed43-fdb0-4490-ab1e-0d3ff281cc94" (UID: "408eed43-fdb0-4490-ab1e-0d3ff281cc94"). InnerVolumeSpecName "kube-api-access-drwbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:16.497346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.497277    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/408eed43-fdb0-4490-ab1e-0d3ff281cc94-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.497346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.497303    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drwbr\" (UniqueName: \"kubernetes.io/projected/408eed43-fdb0-4490-ab1e-0d3ff281cc94-kube-api-access-drwbr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.782138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.782059    5503 scope.go:122] "RemoveContainer" containerID="7be03fa192697573d765fafc0940430373cea5d91fdde600935777e7e3c02f60"
Oct 30 22:30:16.959593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:16.959570    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:30:17.205495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.205462    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerStarted","Data":"2f95e3c6f89e86b17cdd3259415e7b29efad54c771282e8cba15e046b5ebfe4a"}
Oct 30 22:30:17.206928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.206902    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7" event={"ID":"408eed43-fdb0-4490-ab1e-0d3ff281cc94","Type":"ContainerDied","Data":"db9fde917709717590dc7e331e11861cae1a3a7a5d59d264b502e71281593644"}
Oct 30 22:30:17.207054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.206932    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"
Oct 30 22:30:17.207054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.206951    5503 scope.go:122] "RemoveContainer" containerID="8cda84ed74e0926cb4bdcdf92f8c7cdcc7feb65b0833482442c3ef577d17040b"
Oct 30 22:30:17.441323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.441288    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"]
Oct 30 22:30:17.452218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.452189    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29889990-vwtc7"]
Oct 30 22:30:17.790047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.790015    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" path="/var/lib/kubelet/pods/408eed43-fdb0-4490-ab1e-0d3ff281cc94/volumes"
Oct 30 22:30:17.790919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.790896    5503 scope.go:122] "RemoveContainer" containerID="dacb2a33d6ded3c432c5d8d653a05a2b14bab902aaed4faf401276fc16559dda"
Oct 30 22:30:17.791074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.791007    5503 scope.go:122] "RemoveContainer" containerID="12ea40f573228e6103a59939e0afd61165b2b83cc10b58024f09ec67b32d3df9"
Oct 30 22:30:17.791074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.791069    5503 scope.go:122] "RemoveContainer" containerID="2a6eae9feb3bc7b452370e886229d307d93e7bb5cf3a0e98dd54c83e11a2d4c5"
Oct 30 22:30:17.791227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.791206    5503 scope.go:122] "RemoveContainer" containerID="62fada5e93f172a2ca29faf4bbd6e5774d0d16e6b68fa2edb69e14740bf99194"
Oct 30 22:30:17.837841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:17.837800    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:18.029870 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.029757    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:18.218813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.218485    5503 generic.go:356] "Generic (PLEG): container finished" podID="33f5d72e-af25-40a9-8985-91698f555804" containerID="2f95e3c6f89e86b17cdd3259415e7b29efad54c771282e8cba15e046b5ebfe4a" exitCode=60
Oct 30 22:30:18.218813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.218518    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerDied","Data":"2f95e3c6f89e86b17cdd3259415e7b29efad54c771282e8cba15e046b5ebfe4a"}
Oct 30 22:30:18.218813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.218581    5503 scope.go:122] "RemoveContainer" containerID="7be03fa192697573d765fafc0940430373cea5d91fdde600935777e7e3c02f60"
Oct 30 22:30:18.221317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.221282    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerStarted","Data":"429404198a417e6edf10906ee55bb73286a13e99e8e6c80a2395547cc6e21192"}
Oct 30 22:30:18.222336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.222315    5503 scope.go:122] "RemoveContainer" containerID="2f95e3c6f89e86b17cdd3259415e7b29efad54c771282e8cba15e046b5ebfe4a"
Oct 30 22:30:18.222840 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:18.222814    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h_odh-ai-gateway-infra(33f5d72e-af25-40a9-8985-91698f555804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" podUID="33f5d72e-af25-40a9-8985-91698f555804"
Oct 30 22:30:18.368975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.368894    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"]
Oct 30 22:30:18.427211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.427168    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"]
Oct 30 22:30:18.427928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.427909    5503 cpu_manager.go:414] "containerMap: removing container" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.427985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.427932    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.428025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.427988    5503 cpu_manager.go:414] "containerMap: removing container" podUID="fd82aa66-e922-478f-bdcb-a75b3bb5ef86" containerName="revoke-keys"
Oct 30 22:30:18.428025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.427995    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd82aa66-e922-478f-bdcb-a75b3bb5ef86" containerName="revoke-keys"
Oct 30 22:30:18.428025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428021    5503 cpu_manager.go:414] "containerMap: removing container" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.428118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428029    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.428118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428046    5503 cpu_manager.go:414] "containerMap: removing container" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.428118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428054    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.428204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428178    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd82aa66-e922-478f-bdcb-a75b3bb5ef86" containerName="revoke-keys"
Oct 30 22:30:18.428204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428190    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.428204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428201    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.428311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.428211    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="408eed43-fdb0-4490-ab1e-0d3ff281cc94" containerName="cleanup"
Oct 30 22:30:18.433385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.433367    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:18.492212 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.486182    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:18.501654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.499310    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f39eb7-dockercfg-wgrwj\""
Oct 30 22:30:18.501654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.499413    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f39eb7\""
Oct 30 22:30:18.514664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.510698    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"]
Oct 30 22:30:18.547352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.547311    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb257f1-291e-4efa-a34f-27c8df655ea6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:18.547556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.547534    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nm29\" (UniqueName: \"kubernetes.io/projected/dcb257f1-291e-4efa-a34f-27c8df655ea6-kube-api-access-8nm29\") pod \"payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:18.649294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.649201    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm29\" (UniqueName: \"kubernetes.io/projected/dcb257f1-291e-4efa-a34f-27c8df655ea6-kube-api-access-8nm29\") pod \"payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:18.649449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.649355    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb257f1-291e-4efa-a34f-27c8df655ea6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:18.650295 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.650268    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb257f1-291e-4efa-a34f-27c8df655ea6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:18.715423 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.715385    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm29\" (UniqueName: \"kubernetes.io/projected/dcb257f1-291e-4efa-a34f-27c8df655ea6-kube-api-access-8nm29\") pod \"payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:18.749456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:18.749427    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:19.120104 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:19.120075    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb257f1_291e_4efa_a34f_27c8df655ea6.slice/crio-eb9d0effeefefc4ea677643285ddc84c043fa1f0c61673364f1787cb62eaf3a3 WatchSource:0}: Error finding container eb9d0effeefefc4ea677643285ddc84c043fa1f0c61673364f1787cb62eaf3a3: Status 404 returned error can't find the container with id eb9d0effeefefc4ea677643285ddc84c043fa1f0c61673364f1787cb62eaf3a3
Oct 30 22:30:19.127095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.127062    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"]
Oct 30 22:30:19.234482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.234452    5503 generic.go:356] "Generic (PLEG): container finished" podID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerID="429404198a417e6edf10906ee55bb73286a13e99e8e6c80a2395547cc6e21192" exitCode=60
Oct 30 22:30:19.234625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.234529    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerDied","Data":"429404198a417e6edf10906ee55bb73286a13e99e8e6c80a2395547cc6e21192"}
Oct 30 22:30:19.234625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.234590    5503 scope.go:122] "RemoveContainer" containerID="dacb2a33d6ded3c432c5d8d653a05a2b14bab902aaed4faf401276fc16559dda"
Oct 30 22:30:19.235143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.235119    5503 scope.go:122] "RemoveContainer" containerID="429404198a417e6edf10906ee55bb73286a13e99e8e6c80a2395547cc6e21192"
Oct 30 22:30:19.235453 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:19.235426    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk_odh-ai-gateway-infra(003b0aec-651e-43d0-ad83-5cebb42820b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6"
Oct 30 22:30:19.236980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.236957    5503 generic.go:356] "Generic (PLEG): container finished" podID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerID="bac8568184d4e4e93bfac1e5ebbca10e336e4094e94c402b3ae5027e8956e660" exitCode=60
Oct 30 22:30:19.237092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.237028    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" event={"ID":"68ec7968-f189-469a-bd8b-b30ba95bb257","Type":"ContainerDied","Data":"bac8568184d4e4e93bfac1e5ebbca10e336e4094e94c402b3ae5027e8956e660"}
Oct 30 22:30:19.237436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.237414    5503 scope.go:122] "RemoveContainer" containerID="bac8568184d4e4e93bfac1e5ebbca10e336e4094e94c402b3ae5027e8956e660"
Oct 30 22:30:19.237736 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:19.237713    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d_odh-ai-gateway-infra(68ec7968-f189-469a-bd8b-b30ba95bb257)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257"
Oct 30 22:30:19.239287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.239267    5503 generic.go:356] "Generic (PLEG): container finished" podID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerID="0ca0c35be3c7cf7677396890dcf43c4a0726b58dd02ffdb24fcb8f717e5a513f" exitCode=60
Oct 30 22:30:19.239401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.239369    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" event={"ID":"4e1369e2-c33d-49d6-8c4e-76da990030e2","Type":"ContainerDied","Data":"0ca0c35be3c7cf7677396890dcf43c4a0726b58dd02ffdb24fcb8f717e5a513f"}
Oct 30 22:30:19.239762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.239740    5503 scope.go:122] "RemoveContainer" containerID="0ca0c35be3c7cf7677396890dcf43c4a0726b58dd02ffdb24fcb8f717e5a513f"
Oct 30 22:30:19.240005 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:19.239980    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56_odh-ai-gateway-infra(4e1369e2-c33d-49d6-8c4e-76da990030e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2"
Oct 30 22:30:19.245022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.245002    5503 generic.go:356] "Generic (PLEG): container finished" podID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerID="80dc40f019e76c445d686a5bce6963dd59b28164246fa58464633b7ce43615f0" exitCode=60
Oct 30 22:30:19.245122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.245079    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" event={"ID":"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40","Type":"ContainerDied","Data":"80dc40f019e76c445d686a5bce6963dd59b28164246fa58464633b7ce43615f0"}
Oct 30 22:30:19.245472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.245451    5503 scope.go:122] "RemoveContainer" containerID="80dc40f019e76c445d686a5bce6963dd59b28164246fa58464633b7ce43615f0"
Oct 30 22:30:19.245757 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:19.245737    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m57nv_odh-ai-gateway-infra(1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40"
Oct 30 22:30:19.246767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.246747    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" event={"ID":"dcb257f1-291e-4efa-a34f-27c8df655ea6","Type":"ContainerStarted","Data":"3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85"}
Oct 30 22:30:19.246849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.246788    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" event={"ID":"dcb257f1-291e-4efa-a34f-27c8df655ea6","Type":"ContainerStarted","Data":"eb9d0effeefefc4ea677643285ddc84c043fa1f0c61673364f1787cb62eaf3a3"}
Oct 30 22:30:19.246849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.246816    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:19.248292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.248266    5503 scope.go:122] "RemoveContainer" containerID="2a6eae9feb3bc7b452370e886229d307d93e7bb5cf3a0e98dd54c83e11a2d4c5"
Oct 30 22:30:19.264488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.264462    5503 scope.go:122] "RemoveContainer" containerID="12ea40f573228e6103a59939e0afd61165b2b83cc10b58024f09ec67b32d3df9"
Oct 30 22:30:19.278159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.278137    5503 scope.go:122] "RemoveContainer" containerID="62fada5e93f172a2ca29faf4bbd6e5774d0d16e6b68fa2edb69e14740bf99194"
Oct 30 22:30:19.366449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.366421    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"]
Oct 30 22:30:19.372258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.372236    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:19.456768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.456689    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f5d72e-af25-40a9-8985-91698f555804-kube-api-access-vrxtr\" (UniqueName: \"kubernetes.io/projected/33f5d72e-af25-40a9-8985-91698f555804-kube-api-access-vrxtr\") pod \"33f5d72e-af25-40a9-8985-91698f555804\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") "
Oct 30 22:30:19.456917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.456786    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f5d72e-af25-40a9-8985-91698f555804-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f5d72e-af25-40a9-8985-91698f555804-maas-api-service-ca\") pod \"33f5d72e-af25-40a9-8985-91698f555804\" (UID: \"33f5d72e-af25-40a9-8985-91698f555804\") "
Oct 30 22:30:19.457118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.457094    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f5d72e-af25-40a9-8985-91698f555804-maas-api-service-ca" pod "33f5d72e-af25-40a9-8985-91698f555804" (UID: "33f5d72e-af25-40a9-8985-91698f555804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.458796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.458767    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f5d72e-af25-40a9-8985-91698f555804-kube-api-access-vrxtr" pod "33f5d72e-af25-40a9-8985-91698f555804" (UID: "33f5d72e-af25-40a9-8985-91698f555804"). InnerVolumeSpecName "kube-api-access-vrxtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.558023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.557997    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxtr\" (UniqueName: \"kubernetes.io/projected/33f5d72e-af25-40a9-8985-91698f555804-kube-api-access-vrxtr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.558023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.558021    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f5d72e-af25-40a9-8985-91698f555804-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.863488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:19.863423    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" podStartSLOduration=1.863404107 podStartE2EDuration="1.863404107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:19.823168053 +0000 UTC m=+2596.157744823" watchObservedRunningTime="2026-10-30 22:30:19.863404107 +0000 UTC m=+2596.197980880"
Oct 30 22:30:20.255862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.255833    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"
Oct 30 22:30:20.255862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.255840    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h" event={"ID":"33f5d72e-af25-40a9-8985-91698f555804","Type":"ContainerDied","Data":"5bd978c707d55691efbfd586c2785bfbf41d09ec682b3dfcaf288be1b72ff32a"}
Oct 30 22:30:20.256102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.255909    5503 scope.go:122] "RemoveContainer" containerID="2f95e3c6f89e86b17cdd3259415e7b29efad54c771282e8cba15e046b5ebfe4a"
Oct 30 22:30:20.417527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.417498    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:20.473237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.473206    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003b0aec-651e-43d0-ad83-5cebb42820b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003b0aec-651e-43d0-ad83-5cebb42820b6-maas-api-service-ca\") pod \"003b0aec-651e-43d0-ad83-5cebb42820b6\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") "
Oct 30 22:30:20.473375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.473341    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003b0aec-651e-43d0-ad83-5cebb42820b6-kube-api-access-8k2sz\" (UniqueName: \"kubernetes.io/projected/003b0aec-651e-43d0-ad83-5cebb42820b6-kube-api-access-8k2sz\") pod \"003b0aec-651e-43d0-ad83-5cebb42820b6\" (UID: \"003b0aec-651e-43d0-ad83-5cebb42820b6\") "
Oct 30 22:30:20.473550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.473529    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003b0aec-651e-43d0-ad83-5cebb42820b6-maas-api-service-ca" pod "003b0aec-651e-43d0-ad83-5cebb42820b6" (UID: "003b0aec-651e-43d0-ad83-5cebb42820b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.473610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.473599    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/003b0aec-651e-43d0-ad83-5cebb42820b6-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.475615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.475587    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003b0aec-651e-43d0-ad83-5cebb42820b6-kube-api-access-8k2sz" pod "003b0aec-651e-43d0-ad83-5cebb42820b6" (UID: "003b0aec-651e-43d0-ad83-5cebb42820b6"). InnerVolumeSpecName "kube-api-access-8k2sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.482317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.482294    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"]
Oct 30 22:30:20.545058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.544543    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"]
Oct 30 22:30:20.548220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.548192    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29889990-2s88h"]
Oct 30 22:30:20.574608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.574581    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k2sz\" (UniqueName: \"kubernetes.io/projected/003b0aec-651e-43d0-ad83-5cebb42820b6-kube-api-access-8k2sz\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.598994 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.598904    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"]
Oct 30 22:30:20.615345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.615324    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:20.675141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.675113    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1369e2-c33d-49d6-8c4e-76da990030e2-kube-api-access-ljbbd\" (UniqueName: \"kubernetes.io/projected/4e1369e2-c33d-49d6-8c4e-76da990030e2-kube-api-access-ljbbd\") pod \"4e1369e2-c33d-49d6-8c4e-76da990030e2\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") "
Oct 30 22:30:20.675299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.675273    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1369e2-c33d-49d6-8c4e-76da990030e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1369e2-c33d-49d6-8c4e-76da990030e2-maas-api-service-ca\") pod \"4e1369e2-c33d-49d6-8c4e-76da990030e2\" (UID: \"4e1369e2-c33d-49d6-8c4e-76da990030e2\") "
Oct 30 22:30:20.675685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.675655    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1369e2-c33d-49d6-8c4e-76da990030e2-maas-api-service-ca" pod "4e1369e2-c33d-49d6-8c4e-76da990030e2" (UID: "4e1369e2-c33d-49d6-8c4e-76da990030e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.677457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.677436    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1369e2-c33d-49d6-8c4e-76da990030e2-kube-api-access-ljbbd" pod "4e1369e2-c33d-49d6-8c4e-76da990030e2" (UID: "4e1369e2-c33d-49d6-8c4e-76da990030e2"). InnerVolumeSpecName "kube-api-access-ljbbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.727064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.727027    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"]
Oct 30 22:30:20.761922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.761848    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:20.764298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764275    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"]
Oct 30 22:30:20.764737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764724    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:20.764778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764737    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:20.764778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764749    5503 cpu_manager.go:414] "containerMap: removing container" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:20.764778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764756    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:20.764778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764766    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:20.764778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764771    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764781    5503 cpu_manager.go:414] "containerMap: removing container" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764786    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764794    5503 cpu_manager.go:414] "containerMap: removing container" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764799    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764809    5503 cpu_manager.go:414] "containerMap: removing container" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764814    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764819    5503 cpu_manager.go:414] "containerMap: removing container" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764823    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764836    5503 cpu_manager.go:414] "containerMap: removing container" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764840    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764849    5503 cpu_manager.go:414] "containerMap: removing container" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764853    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764871    5503 cpu_manager.go:414] "containerMap: removing container" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.764935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764875    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764957    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764964    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764972    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764977    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764982    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764989    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.764996    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.765003    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.765008    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f5d72e-af25-40a9-8985-91698f555804" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.765015    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.765337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.765019    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" containerName="cleanup"
Oct 30 22:30:20.768236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.768212    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.776365 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.776342    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1369e2-c33d-49d6-8c4e-76da990030e2-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.776365 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.776364    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbbd\" (UniqueName: \"kubernetes.io/projected/4e1369e2-c33d-49d6-8c4e-76da990030e2-kube-api-access-ljbbd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.827910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.827880    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-28f124-openshift-default-dockercfg-q9xb5\""
Oct 30 22:30:20.849567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.849529    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"]
Oct 30 22:30:20.876163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.876138    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:20.877201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.877180    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-maas-api-service-ca\") pod \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") "
Oct 30 22:30:20.877342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.877226    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-kube-api-access-8pmqx\" (UniqueName: \"kubernetes.io/projected/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-kube-api-access-8pmqx\") pod \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\" (UID: \"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40\") "
Oct 30 22:30:20.877551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.877527    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-maas-api-service-ca" pod "1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" (UID: "1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.877627 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.877569    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-token\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.877700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.877630    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istiod-ca-cert\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878224    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-envoy\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878276    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfrt\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-kube-api-access-nbfrt\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878374    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878436    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-credential-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878579    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-podinfo\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878658    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-certs\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878711    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-data\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.878905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.878880    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.890672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.879927    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-kube-api-access-8pmqx" pod "1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" (UID: "1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40"). InnerVolumeSpecName "kube-api-access-8pmqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.980037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980009    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ec7968-f189-469a-bd8b-b30ba95bb257-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec7968-f189-469a-bd8b-b30ba95bb257-maas-api-service-ca\") pod \"68ec7968-f189-469a-bd8b-b30ba95bb257\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") "
Oct 30 22:30:20.980217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980046    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ec7968-f189-469a-bd8b-b30ba95bb257-kube-api-access-2vlgd\" (UniqueName: \"kubernetes.io/projected/68ec7968-f189-469a-bd8b-b30ba95bb257-kube-api-access-2vlgd\") pod \"68ec7968-f189-469a-bd8b-b30ba95bb257\" (UID: \"68ec7968-f189-469a-bd8b-b30ba95bb257\") "
Oct 30 22:30:20.980217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980195    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-token\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.980294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980252    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istiod-ca-cert\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.980329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980296    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-envoy\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.980329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980322    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfrt\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-kube-api-access-nbfrt\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.980450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980333    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ec7968-f189-469a-bd8b-b30ba95bb257-maas-api-service-ca" pod "68ec7968-f189-469a-bd8b-b30ba95bb257" (UID: "68ec7968-f189-469a-bd8b-b30ba95bb257"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.980450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980417    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.980867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980841    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980894    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-credential-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980949    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-podinfo\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.980983    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-certs\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.981015    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istiod-ca-cert\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.981021    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-data\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.981177    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmqx\" (UniqueName: \"kubernetes.io/projected/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40-kube-api-access-8pmqx\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.981271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.981198    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec7968-f189-469a-bd8b-b30ba95bb257-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.981398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.981272    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-credential-socket\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.981405    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-certs\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.981508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.981453    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-data\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.982392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.982366    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ec7968-f189-469a-bd8b-b30ba95bb257-kube-api-access-2vlgd" pod "68ec7968-f189-469a-bd8b-b30ba95bb257" (UID: "68ec7968-f189-469a-bd8b-b30ba95bb257"). InnerVolumeSpecName "kube-api-access-2vlgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.982689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.982670    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-envoy\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:20.983227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:20.983207    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-podinfo\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:21.043182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.043102    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-token\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:21.043602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.043580    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfrt\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-kube-api-access-nbfrt\") pod \"e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") " pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:21.082400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.082369    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:21.082851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.082828    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vlgd\" (UniqueName: \"kubernetes.io/projected/68ec7968-f189-469a-bd8b-b30ba95bb257-kube-api-access-2vlgd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.278966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.278940    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"
Oct 30 22:30:21.279171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.278959    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk" event={"ID":"003b0aec-651e-43d0-ad83-5cebb42820b6","Type":"ContainerDied","Data":"42e6b5be6c8c40dfae10735c9ca7e429b013b7b37f5a06b536837e3fcb6f1f54"}
Oct 30 22:30:21.279171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.279053    5503 scope.go:122] "RemoveContainer" containerID="429404198a417e6edf10906ee55bb73286a13e99e8e6c80a2395547cc6e21192"
Oct 30 22:30:21.280760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.280609    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"
Oct 30 22:30:21.280760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.280605    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d" event={"ID":"68ec7968-f189-469a-bd8b-b30ba95bb257","Type":"ContainerDied","Data":"7fe48e20daf2dfbf97c18f815684802c9cd40f0080469266ee567bb478f9f435"}
Oct 30 22:30:21.282995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.282974    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56" event={"ID":"4e1369e2-c33d-49d6-8c4e-76da990030e2","Type":"ContainerDied","Data":"27eef3909fdd42b8efb06d64eec47c31292fb5dabaa5952d135d6aee3f019354"}
Oct 30 22:30:21.283124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.283078    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"
Oct 30 22:30:21.289173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.289147    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv" event={"ID":"1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40","Type":"ContainerDied","Data":"3943d4d8cf7fbde383ed1c76cf3d224a6a476c1cbc775dee31b9f4493b85fe1d"}
Oct 30 22:30:21.289285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.289186    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"
Oct 30 22:30:21.292199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.292018    5503 scope.go:122] "RemoveContainer" containerID="bac8568184d4e4e93bfac1e5ebbca10e336e4094e94c402b3ae5027e8956e660"
Oct 30 22:30:21.304006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.303991    5503 scope.go:122] "RemoveContainer" containerID="0ca0c35be3c7cf7677396890dcf43c4a0726b58dd02ffdb24fcb8f717e5a513f"
Oct 30 22:30:21.312506 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.312488    5503 scope.go:122] "RemoveContainer" containerID="80dc40f019e76c445d686a5bce6963dd59b28164246fa58464633b7ce43615f0"
Oct 30 22:30:21.502574 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:21.502546    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdddc2d_a58b_44b0_afed_cc12f2018c75.slice/crio-bd6fb75bae0565260fb10c3adaeca9e85bccde0af53327570f82a113161ba223 WatchSource:0}: Error finding container bd6fb75bae0565260fb10c3adaeca9e85bccde0af53327570f82a113161ba223: Status 404 returned error can't find the container with id bd6fb75bae0565260fb10c3adaeca9e85bccde0af53327570f82a113161ba223
Oct 30 22:30:21.505189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.505150    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:30:21.505311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.505241    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:30:21.505311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.505296    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:30:21.505875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.505855    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"]
Oct 30 22:30:21.632677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.632619    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"]
Oct 30 22:30:21.635879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.635852    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29889990-hztxk"]
Oct 30 22:30:21.751130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.751105    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"]
Oct 30 22:30:21.755931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.755910    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-459aa8-29889990-jxf56"]
Oct 30 22:30:21.788582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.788556    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" path="/var/lib/kubelet/pods/003b0aec-651e-43d0-ad83-5cebb42820b6/volumes"
Oct 30 22:30:21.789001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.788987    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f5d72e-af25-40a9-8985-91698f555804" path="/var/lib/kubelet/pods/33f5d72e-af25-40a9-8985-91698f555804/volumes"
Oct 30 22:30:21.789484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.789458    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" path="/var/lib/kubelet/pods/4e1369e2-c33d-49d6-8c4e-76da990030e2/volumes"
Oct 30 22:30:21.868715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.868603    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"]
Oct 30 22:30:21.876032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:21.876007    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m57nv"]
Oct 30 22:30:22.002377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:22.002346    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"]
Oct 30 22:30:22.005199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:22.005168    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7f66ab-29889990-vck4d"]
Oct 30 22:30:22.303273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:22.303235    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" event={"ID":"3fdddc2d-a58b-44b0-afed-cc12f2018c75","Type":"ContainerStarted","Data":"dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c"}
Oct 30 22:30:22.303459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:22.303288    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" event={"ID":"3fdddc2d-a58b-44b0-afed-cc12f2018c75","Type":"ContainerStarted","Data":"bd6fb75bae0565260fb10c3adaeca9e85bccde0af53327570f82a113161ba223"}
Oct 30 22:30:23.082956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:23.082929    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:23.084367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:23.084336    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:30:23.084521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:23.084391    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:30:23.530297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:23.530242    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:23.796485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:23.796403    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40" path="/var/lib/kubelet/pods/1858e3e4-bdfe-4ba9-91d4-f79e2fad5f40/volumes"
Oct 30 22:30:23.796867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:23.796852    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" path="/var/lib/kubelet/pods/68ec7968-f189-469a-bd8b-b30ba95bb257/volumes"
Oct 30 22:30:24.083333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.083250    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:30:24.083333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.083310    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:30:24.439663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.436250    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:24.746777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.746683    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podStartSLOduration=4.746670757 podStartE2EDuration="4.746670757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:22.440026765 +0000 UTC m=+2598.774603573" watchObservedRunningTime="2026-10-30 22:30:24.746670757 +0000 UTC m=+2601.081247526"
Oct 30 22:30:24.749739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.749715    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"]
Oct 30 22:30:24.750182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750169    5503 cpu_manager.go:414] "containerMap: removing container" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:24.750232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750184    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:24.750232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750199    5503 cpu_manager.go:414] "containerMap: removing container" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:24.750232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750205    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:24.750329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750236    5503 cpu_manager.go:414] "containerMap: removing container" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:24.750329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750241    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:24.750329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750247    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:24.750329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750252    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1369e2-c33d-49d6-8c4e-76da990030e2" containerName="cleanup"
Oct 30 22:30:24.750329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750329    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="003b0aec-651e-43d0-ad83-5cebb42820b6" containerName="cleanup"
Oct 30 22:30:24.750479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750336    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:24.750479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750343    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:24.750479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.750352    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:24.755466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.755447    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.808529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.808495    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"]
Oct 30 22:30:24.849680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.849616    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.849854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.849746    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.849909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.849885    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2gc\" (UniqueName: \"kubernetes.io/projected/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-kube-api-access-bf2gc\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.855699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.855624    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:24.855699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.855624    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:24.855699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.855683    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:24.860571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.860553    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:30:24.951376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.951333    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2gc\" (UniqueName: \"kubernetes.io/projected/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-kube-api-access-bf2gc\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.951573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.951487    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.951710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.951582    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.952261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.952238    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:24.953889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:24.953867    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:25.014372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:25.014302    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2gc\" (UniqueName: \"kubernetes.io/projected/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-kube-api-access-bf2gc\") pod \"maas-api-revoke-sub-dc8c1677-s796n\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:25.068568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:25.068538    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:25.083776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:25.083753    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:30:25.084034 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:25.083826    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:30:25.464537 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:25.464507    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b114d3c_7249_4ec6_9ae9_36e8d1a1e735.slice/crio-6380e69d80dfd16483f27c668f0ff79a9d8f6df915233aeba30a7702de21994e WatchSource:0}: Error finding container 6380e69d80dfd16483f27c668f0ff79a9d8f6df915233aeba30a7702de21994e: Status 404 returned error can't find the container with id 6380e69d80dfd16483f27c668f0ff79a9d8f6df915233aeba30a7702de21994e
Oct 30 22:30:25.468024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:25.467999    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"]
Oct 30 22:30:25.601803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:25.601766    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:26.083436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:26.083404    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:30:26.083613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:26.083459    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:30:26.355980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:26.355945    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n" event={"ID":"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735","Type":"ContainerStarted","Data":"79e23aea44949a0b3438ed5e730d1aae186c31a2a42975a504410b47a0afa646"}
Oct 30 22:30:26.356374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:26.355999    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n" event={"ID":"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735","Type":"ContainerStarted","Data":"6380e69d80dfd16483f27c668f0ff79a9d8f6df915233aeba30a7702de21994e"}
Oct 30 22:30:26.477044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:26.476988    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n" podStartSLOduration=2.476969916 podStartE2EDuration="2.476969916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:26.475145315 +0000 UTC m=+2602.809722083" watchObservedRunningTime="2026-10-30 22:30:26.476969916 +0000 UTC m=+2602.811546685"
Oct 30 22:30:27.083137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:27.083097    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:30:27.083327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:27.083151    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:30:27.365110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:27.365011    5503 generic.go:356] "Generic (PLEG): container finished" podID="2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" containerID="79e23aea44949a0b3438ed5e730d1aae186c31a2a42975a504410b47a0afa646" exitCode=0
Oct 30 22:30:27.365575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:27.365100    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n" event={"ID":"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735","Type":"ContainerDied","Data":"79e23aea44949a0b3438ed5e730d1aae186c31a2a42975a504410b47a0afa646"}
Oct 30 22:30:28.087723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.087694    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:28.088154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.088133    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:28.088951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.088930    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:30:28.099540 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:28.099512    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:30:28.189412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.189374    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:28.515914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.515888    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:28.592887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.592854    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-kube-api-access-bf2gc\" (UniqueName: \"kubernetes.io/projected/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-kube-api-access-bf2gc\") pod \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") "
Oct 30 22:30:28.593071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.592915    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-openshift-ca\") pod \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") "
Oct 30 22:30:28.593071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.592958    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-xks-ca\") pod \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\" (UID: \"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735\") "
Oct 30 22:30:28.593366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.593333    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-openshift-ca" pod "2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" (UID: "2b114d3c-7249-4ec6-9ae9-36e8d1a1e735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:28.594946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.594920    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-xks-ca" pod "2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" (UID: "2b114d3c-7249-4ec6-9ae9-36e8d1a1e735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:28.595042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.595010    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-kube-api-access-bf2gc" pod "2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" (UID: "2b114d3c-7249-4ec6-9ae9-36e8d1a1e735"). InnerVolumeSpecName "kube-api-access-bf2gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:28.693957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.693916    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf2gc\" (UniqueName: \"kubernetes.io/projected/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-kube-api-access-bf2gc\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:28.693957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.693952    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:28.693957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.693962    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:28.797035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:28.797002    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:29.012358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.012244    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"]
Oct 30 22:30:29.012933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.012913    5503 cpu_manager.go:414] "containerMap: removing container" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:29.012993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.012937    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ec7968-f189-469a-bd8b-b30ba95bb257" containerName="cleanup"
Oct 30 22:30:29.012993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.012956    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" containerName="revoke-keys"
Oct 30 22:30:29.012993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.012964    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" containerName="revoke-keys"
Oct 30 22:30:29.013130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.013116    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" containerName="revoke-keys"
Oct 30 22:30:29.016464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.016449    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.074849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.074815    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"]
Oct 30 22:30:29.097728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.097695    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.097883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.097755    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.097883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.097844    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.097883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.097880    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxz7c\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-kube-api-access-vxz7c\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.098017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.097949    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.098053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.098016    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.098096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.098061    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.098153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.098137    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.098251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.098194    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199074    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199123    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199153    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199176    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxz7c\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-kube-api-access-vxz7c\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199214    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199285    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199336    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199380    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199474    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-credential-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199502    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199531    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-data\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199897    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-certs\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.199993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.199921    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-socket\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.200059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.200039    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istiod-ca-cert\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.201665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.201618    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-envoy\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.201865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.201847    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-podinfo\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.261826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.261792    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-token\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.261977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.261909    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxz7c\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-kube-api-access-vxz7c\") pod \"e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") " pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.329813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.329741    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:29.385190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.385097    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"
Oct 30 22:30:29.391533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.388808    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n" event={"ID":"2b114d3c-7249-4ec6-9ae9-36e8d1a1e735","Type":"ContainerDied","Data":"6380e69d80dfd16483f27c668f0ff79a9d8f6df915233aeba30a7702de21994e"}
Oct 30 22:30:29.391533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.388868    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6380e69d80dfd16483f27c668f0ff79a9d8f6df915233aeba30a7702de21994e"
Oct 30 22:30:29.754802 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:29.754764    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cf9753_70e3_4c77_84ca_688b9ef58cb9.slice/crio-050db5803f8a488f9fe3887d2d21f1352a3cdbf7e1804cbed40cbd3f4e417d5e WatchSource:0}: Error finding container 050db5803f8a488f9fe3887d2d21f1352a3cdbf7e1804cbed40cbd3f4e417d5e: Status 404 returned error can't find the container with id 050db5803f8a488f9fe3887d2d21f1352a3cdbf7e1804cbed40cbd3f4e417d5e
Oct 30 22:30:29.757968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.757928    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:30:29.758060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.758009    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:30:29.758060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.758053    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:30:29.758701 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:29.758683    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"]
Oct 30 22:30:30.275390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.275352    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:30:30.393079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.393043    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" event={"ID":"77cf9753-70e3-4c77-84ca-688b9ef58cb9","Type":"ContainerStarted","Data":"000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a"}
Oct 30 22:30:30.393436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.393098    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" event={"ID":"77cf9753-70e3-4c77-84ca-688b9ef58cb9","Type":"ContainerStarted","Data":"050db5803f8a488f9fe3887d2d21f1352a3cdbf7e1804cbed40cbd3f4e417d5e"}
Oct 30 22:30:30.532727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.531478    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podStartSLOduration=2.531457418 podStartE2EDuration="2.531457418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:30.519253033 +0000 UTC m=+2606.853829815" watchObservedRunningTime="2026-10-30 22:30:30.531457418 +0000 UTC m=+2606.866034189"
Oct 30 22:30:30.643446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.643389    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"]
Oct 30 22:30:30.647887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.647862    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.685937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.685900    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"]
Oct 30 22:30:30.701332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.701304    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:30:30.701508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.701359    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:30:30.701508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.701468    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:30.701614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.701565    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:30:30.738732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.738307    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjpt\" (UniqueName: \"kubernetes.io/projected/750db2ca-793c-4e63-b58e-25b52f0d21bf-kube-api-access-jpjpt\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.738732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.738451    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.738732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.738527    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.841717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.841302    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.841717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.841414    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.841717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.841570    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjpt\" (UniqueName: \"kubernetes.io/projected/750db2ca-793c-4e63-b58e-25b52f0d21bf-kube-api-access-jpjpt\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.843476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.843292    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.845625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.845597    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.905747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.905713    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjpt\" (UniqueName: \"kubernetes.io/projected/750db2ca-793c-4e63-b58e-25b52f0d21bf-kube-api-access-jpjpt\") pod \"maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:30.964031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:30.963996    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:31.334050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:31.331964    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:31.341062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:31.341029    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:30:31.341234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:31.341092    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:30:31.360100 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:31.360073    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750db2ca_793c_4e63_b58e_25b52f0d21bf.slice/crio-8a2eed6115a158aca6f8953a4f9dd1f1bd2e2282d73dbb31b99fa363e6aceabf WatchSource:0}: Error finding container 8a2eed6115a158aca6f8953a4f9dd1f1bd2e2282d73dbb31b99fa363e6aceabf: Status 404 returned error can't find the container with id 8a2eed6115a158aca6f8953a4f9dd1f1bd2e2282d73dbb31b99fa363e6aceabf
Oct 30 22:30:31.364448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:31.364423    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"]
Oct 30 22:30:31.400797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:31.400768    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj" event={"ID":"750db2ca-793c-4e63-b58e-25b52f0d21bf","Type":"ContainerStarted","Data":"8a2eed6115a158aca6f8953a4f9dd1f1bd2e2282d73dbb31b99fa363e6aceabf"}
Oct 30 22:30:32.340080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:32.340038    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:30:32.340595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:32.340105    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:30:32.408579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:32.408552    5503 generic.go:356] "Generic (PLEG): container finished" podID="750db2ca-793c-4e63-b58e-25b52f0d21bf" containerID="28b542a7d750fedadce33c118a568777f93e38d88bd140ceb31c35dc08f0af77" exitCode=0
Oct 30 22:30:32.408737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:32.408653    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj" event={"ID":"750db2ca-793c-4e63-b58e-25b52f0d21bf","Type":"ContainerDied","Data":"28b542a7d750fedadce33c118a568777f93e38d88bd140ceb31c35dc08f0af77"}
Oct 30 22:30:33.331994 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.331958    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:30:33.332270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.332241    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:30:33.385935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.385905    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:33.684384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.684355    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:33.773779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.773746    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-openshift-ca\") pod \"750db2ca-793c-4e63-b58e-25b52f0d21bf\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") "
Oct 30 22:30:33.773955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.773834    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-xks-ca\") pod \"750db2ca-793c-4e63-b58e-25b52f0d21bf\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") "
Oct 30 22:30:33.773955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.773872    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750db2ca-793c-4e63-b58e-25b52f0d21bf-kube-api-access-jpjpt\" (UniqueName: \"kubernetes.io/projected/750db2ca-793c-4e63-b58e-25b52f0d21bf-kube-api-access-jpjpt\") pod \"750db2ca-793c-4e63-b58e-25b52f0d21bf\" (UID: \"750db2ca-793c-4e63-b58e-25b52f0d21bf\") "
Oct 30 22:30:33.774164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.774136    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-openshift-ca" pod "750db2ca-793c-4e63-b58e-25b52f0d21bf" (UID: "750db2ca-793c-4e63-b58e-25b52f0d21bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:33.776017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.775984    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-xks-ca" pod "750db2ca-793c-4e63-b58e-25b52f0d21bf" (UID: "750db2ca-793c-4e63-b58e-25b52f0d21bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:33.776140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.776117    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750db2ca-793c-4e63-b58e-25b52f0d21bf-kube-api-access-jpjpt" pod "750db2ca-793c-4e63-b58e-25b52f0d21bf" (UID: "750db2ca-793c-4e63-b58e-25b52f0d21bf"). InnerVolumeSpecName "kube-api-access-jpjpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:33.875113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.875080    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.875113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.875115    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750db2ca-793c-4e63-b58e-25b52f0d21bf-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:33.875337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:33.875131    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjpt\" (UniqueName: \"kubernetes.io/projected/750db2ca-793c-4e63-b58e-25b52f0d21bf-kube-api-access-jpjpt\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:34.344679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:34.341068    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:30:34.344679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:34.341175    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:30:34.447299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:34.447263    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj" event={"ID":"750db2ca-793c-4e63-b58e-25b52f0d21bf","Type":"ContainerDied","Data":"8a2eed6115a158aca6f8953a4f9dd1f1bd2e2282d73dbb31b99fa363e6aceabf"}
Oct 30 22:30:34.447299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:34.447297    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"
Oct 30 22:30:34.447765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:34.447336    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a2eed6115a158aca6f8953a4f9dd1f1bd2e2282d73dbb31b99fa363e6aceabf"
Oct 30 22:30:35.339662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:35.338691    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:30:35.339662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:35.338770    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:30:35.872385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:35.872352    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.330709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:36.330669    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:30:36.330957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:36.330720    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:30:36.625283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:36.625174    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:36.790660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:36.788121    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:37.058290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:37.058258    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:30:37.335780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:37.335711    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:37.336196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:37.336177    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:37.337057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:37.337039    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:30:38.428277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:38.428242    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:39.861768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:39.861718    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:41.065752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:41.065718    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.230313 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:41.230270    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:41.398291 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:41.398218    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:43.297036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:43.296995    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:43.448205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:43.448171    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:44.305026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:44.304993    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.055910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.055878    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.241276 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.240791    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:47.473188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.473157    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"]
Oct 30 22:30:47.473904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.473882    5503 cpu_manager.go:414] "containerMap: removing container" podUID="750db2ca-793c-4e63-b58e-25b52f0d21bf" containerName="revoke-keys"
Oct 30 22:30:47.473966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.473908    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750db2ca-793c-4e63-b58e-25b52f0d21bf" containerName="revoke-keys"
Oct 30 22:30:47.474108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.474095    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="750db2ca-793c-4e63-b58e-25b52f0d21bf" containerName="revoke-keys"
Oct 30 22:30:47.479038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.479016    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:47.534355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.534325    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54e2f66-cb0e-41e5-af46-3fb7948dd396-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:47.534535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.534450    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlz7\" (UniqueName: \"kubernetes.io/projected/a54e2f66-cb0e-41e5-af46-3fb7948dd396-kube-api-access-7vlz7\") pod \"payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:47.538079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.538049    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"]
Oct 30 22:30:47.607032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.607000    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fc6466-dockercfg-lnth8\""
Oct 30 22:30:47.608827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.608796    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fc6466\""
Oct 30 22:30:47.637505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.637455    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54e2f66-cb0e-41e5-af46-3fb7948dd396-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:47.658856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.658816    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlz7\" (UniqueName: \"kubernetes.io/projected/a54e2f66-cb0e-41e5-af46-3fb7948dd396-kube-api-access-7vlz7\") pod \"payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:47.659681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.642263    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54e2f66-cb0e-41e5-af46-3fb7948dd396-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:47.722296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.722261    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlz7\" (UniqueName: \"kubernetes.io/projected/a54e2f66-cb0e-41e5-af46-3fb7948dd396-kube-api-access-7vlz7\") pod \"payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:47.792735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:47.792659    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:48.164631 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:30:48.164593    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54e2f66_cb0e_41e5_af46_3fb7948dd396.slice/crio-fc87a4fcfb54d5b09524b632c1efe4b9d025b1d796f7b33ccfb93065a08a5135 WatchSource:0}: Error finding container fc87a4fcfb54d5b09524b632c1efe4b9d025b1d796f7b33ccfb93065a08a5135: Status 404 returned error can't find the container with id fc87a4fcfb54d5b09524b632c1efe4b9d025b1d796f7b33ccfb93065a08a5135
Oct 30 22:30:48.168926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:48.168905    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"]
Oct 30 22:30:48.657413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:48.657377    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" event={"ID":"a54e2f66-cb0e-41e5-af46-3fb7948dd396","Type":"ContainerStarted","Data":"654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06"}
Oct 30 22:30:48.657573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:48.657453    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" event={"ID":"a54e2f66-cb0e-41e5-af46-3fb7948dd396","Type":"ContainerStarted","Data":"fc87a4fcfb54d5b09524b632c1efe4b9d025b1d796f7b33ccfb93065a08a5135"}
Oct 30 22:30:48.657573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:48.657491    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:30:48.784998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:48.784946    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" podStartSLOduration=1.784930722 podStartE2EDuration="1.784930722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:48.78261858 +0000 UTC m=+2625.117195349" watchObservedRunningTime="2026-10-30 22:30:48.784930722 +0000 UTC m=+2625.119507490"
Oct 30 22:30:49.151181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:49.151150    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:51.461279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:51.461239    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:53.907985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:53.907947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.067508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:54.067463    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:54.275181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:54.275080    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:56.960373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:56.960341    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:58.098578 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:30:58.098544    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:30:58.729844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:58.729810    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:30:59.667558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:30:59.667525    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:31:02.803843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:02.803798    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.247477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:04.247444    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.401015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:04.400983    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.559549 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:04.559464    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.712989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:04.712954    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:04.717815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:04.717792    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"]
Oct 30 22:31:04.733699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:04.733673    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-459aa8-6d5pj"]
Oct 30 22:31:05.056537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:05.056507    5503 scope.go:122] "RemoveContainer" containerID="626bd6e815cd30b7f4c985253cc8d777bf9da2311ed09d32439a24d2b85cdcb8"
Oct 30 22:31:05.218526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:05.218493    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:05.788882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:05.788848    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750db2ca-793c-4e63-b58e-25b52f0d21bf" path="/var/lib/kubelet/pods/750db2ca-793c-4e63-b58e-25b52f0d21bf/volumes"
Oct 30 22:31:08.184913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:08.184876    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:31:09.442859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:09.442800    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:11.464836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:11.464806    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:11.679880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:11.679841    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:11.850570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:11.850485    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:12.736147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.736111    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"]
Oct 30 22:31:12.742335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.742311    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.798424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.798393    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"]
Oct 30 22:31:12.844598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.844570    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:31:12.844598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.844584    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:31:12.850746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.850723    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:31:12.850874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.850766    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:31:12.858884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.858858    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204d1574-3f88-462a-8522-60af3d24338c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.859071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.859044    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204d1574-3f88-462a-8522-60af3d24338c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.859254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.859239    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57spt\" (UniqueName: \"kubernetes.io/projected/204d1574-3f88-462a-8522-60af3d24338c-kube-api-access-57spt\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.960495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.960458    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57spt\" (UniqueName: \"kubernetes.io/projected/204d1574-3f88-462a-8522-60af3d24338c-kube-api-access-57spt\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.960742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.960560    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204d1574-3f88-462a-8522-60af3d24338c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.960742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.960583    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204d1574-3f88-462a-8522-60af3d24338c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.961290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.961260    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204d1574-3f88-462a-8522-60af3d24338c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:12.962873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:12.962856    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204d1574-3f88-462a-8522-60af3d24338c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:13.030908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:13.030850    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57spt\" (UniqueName: \"kubernetes.io/projected/204d1574-3f88-462a-8522-60af3d24338c-kube-api-access-57spt\") pod \"maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:13.056740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:13.056718    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:13.426350 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:31:13.426314    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204d1574_3f88_462a_8522_60af3d24338c.slice/crio-827f7a435d90d796a72317e885e4e43747cfcec3c9ce109395b64a2869e8007c WatchSource:0}: Error finding container 827f7a435d90d796a72317e885e4e43747cfcec3c9ce109395b64a2869e8007c: Status 404 returned error can't find the container with id 827f7a435d90d796a72317e885e4e43747cfcec3c9ce109395b64a2869e8007c
Oct 30 22:31:13.430228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:13.430206    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"]
Oct 30 22:31:13.849802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:13.849762    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z" event={"ID":"204d1574-3f88-462a-8522-60af3d24338c","Type":"ContainerStarted","Data":"d895cfeebd94c4e766ebc9ace8c4bd2406d6193839309303f25fdc5eaf07e818"}
Oct 30 22:31:13.850228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:13.849823    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z" event={"ID":"204d1574-3f88-462a-8522-60af3d24338c","Type":"ContainerStarted","Data":"827f7a435d90d796a72317e885e4e43747cfcec3c9ce109395b64a2869e8007c"}
Oct 30 22:31:13.969148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:13.969094    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z" podStartSLOduration=1.969079665 podStartE2EDuration="1.969079665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:13.966888921 +0000 UTC m=+2650.301465712" watchObservedRunningTime="2026-10-30 22:31:13.969079665 +0000 UTC m=+2650.303656438"
Oct 30 22:31:14.382040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:14.382010    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:14.858190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:14.858152    5503 generic.go:356] "Generic (PLEG): container finished" podID="204d1574-3f88-462a-8522-60af3d24338c" containerID="d895cfeebd94c4e766ebc9ace8c4bd2406d6193839309303f25fdc5eaf07e818" exitCode=0
Oct 30 22:31:14.858720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:14.858260    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z" event={"ID":"204d1574-3f88-462a-8522-60af3d24338c","Type":"ContainerDied","Data":"d895cfeebd94c4e766ebc9ace8c4bd2406d6193839309303f25fdc5eaf07e818"}
Oct 30 22:31:15.773191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:15.773159    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:15.931057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:15.931023    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:16.015204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.015179    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:16.195350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.195318    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204d1574-3f88-462a-8522-60af3d24338c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204d1574-3f88-462a-8522-60af3d24338c-maas-api-xks-ca\") pod \"204d1574-3f88-462a-8522-60af3d24338c\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") "
Oct 30 22:31:16.195536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.195390    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204d1574-3f88-462a-8522-60af3d24338c-kube-api-access-57spt\" (UniqueName: \"kubernetes.io/projected/204d1574-3f88-462a-8522-60af3d24338c-kube-api-access-57spt\") pod \"204d1574-3f88-462a-8522-60af3d24338c\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") "
Oct 30 22:31:16.195536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.195533    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204d1574-3f88-462a-8522-60af3d24338c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204d1574-3f88-462a-8522-60af3d24338c-maas-api-openshift-ca\") pod \"204d1574-3f88-462a-8522-60af3d24338c\" (UID: \"204d1574-3f88-462a-8522-60af3d24338c\") "
Oct 30 22:31:16.195952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.195921    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204d1574-3f88-462a-8522-60af3d24338c-maas-api-openshift-ca" pod "204d1574-3f88-462a-8522-60af3d24338c" (UID: "204d1574-3f88-462a-8522-60af3d24338c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:16.197497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.197469    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204d1574-3f88-462a-8522-60af3d24338c-maas-api-xks-ca" pod "204d1574-3f88-462a-8522-60af3d24338c" (UID: "204d1574-3f88-462a-8522-60af3d24338c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:16.197606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.197506    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204d1574-3f88-462a-8522-60af3d24338c-kube-api-access-57spt" pod "204d1574-3f88-462a-8522-60af3d24338c" (UID: "204d1574-3f88-462a-8522-60af3d24338c"). InnerVolumeSpecName "kube-api-access-57spt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.297116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.297087    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57spt\" (UniqueName: \"kubernetes.io/projected/204d1574-3f88-462a-8522-60af3d24338c-kube-api-access-57spt\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.297116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.297114    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204d1574-3f88-462a-8522-60af3d24338c-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.297288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.297128    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204d1574-3f88-462a-8522-60af3d24338c-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.873795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.873760    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"
Oct 30 22:31:16.873795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.873785    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z" event={"ID":"204d1574-3f88-462a-8522-60af3d24338c","Type":"ContainerDied","Data":"827f7a435d90d796a72317e885e4e43747cfcec3c9ce109395b64a2869e8007c"}
Oct 30 22:31:16.874065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:16.873830    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="827f7a435d90d796a72317e885e4e43747cfcec3c9ce109395b64a2869e8007c"
Oct 30 22:31:19.042816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:19.042782    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:23.793893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:23.793859    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.647210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:24.647175    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:24.799134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:24.799095    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:25.955967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:25.955938    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:28.099705 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:31:28.099669    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:31:28.426830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:28.426801    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:32.924444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:32.924404    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:35.972487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:35.972457    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:31:37.489854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:37.489819    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:37.646992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:37.646947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:37.840383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:37.840126    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:42.341046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:42.341014    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:46.879303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:46.879269    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:51.504655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:51.504596    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:56.141171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:56.141111    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:58.098656 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:31:58.098597    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:31:58.591515 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:58.591481    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:59.097474 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:59.097436    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:31:59.755140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:31:59.755103    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:00.255333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:00.247745    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:01.376188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:01.376155    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:02.774219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:02.774171    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:03.833862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:03.833832    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:03.989040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:03.989006    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:04.138294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:04.138197    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:04.166937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:04.166908    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:32:04.212496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:04.212466    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:32:05.113368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:05.113337    5503 scope.go:122] "RemoveContainer" containerID="62a2b5e212cac7d5465261c99fadec6df832e9cda8776a6406bde2c01f666391"
Oct 30 22:32:05.123118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:05.123101    5503 scope.go:122] "RemoveContainer" containerID="c43ad0c127df6505aa8f9ac72b42d15626254012f1daf0ff5322ffb80e459119"
Oct 30 22:32:05.132493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:05.132473    5503 scope.go:122] "RemoveContainer" containerID="2e6e8a4106a9fea4b1e820a6cdfefb6244bc07349329f4821769066968357999"
Oct 30 22:32:05.141529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:05.141509    5503 scope.go:122] "RemoveContainer" containerID="72a2c8863e21ad78d6894fe3d8fa1463ac702074ecc98005c514c59adb272024"
Oct 30 22:32:05.150462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:05.150445    5503 scope.go:122] "RemoveContainer" containerID="f4f99cd6667a985b5f66f100fc0b2d5a8881499a2f81f39467cc10d5b1df10f3"
Oct 30 22:32:06.558519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:06.558485    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:07.093071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.093023    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:07.278358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.278321    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:07.510726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.510619    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"]
Oct 30 22:32:07.511720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.511692    5503 cpu_manager.go:414] "containerMap: removing container" podUID="204d1574-3f88-462a-8522-60af3d24338c" containerName="revoke-keys"
Oct 30 22:32:07.511720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.511720    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204d1574-3f88-462a-8522-60af3d24338c" containerName="revoke-keys"
Oct 30 22:32:07.512081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.512065    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="204d1574-3f88-462a-8522-60af3d24338c" containerName="revoke-keys"
Oct 30 22:32:07.518903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.518868    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.565127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.565090    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"]
Oct 30 22:32:07.628214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.628168    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxxk\" (UniqueName: \"kubernetes.io/projected/fb8bafda-7d41-4661-800f-d2daf4f40a8b-kube-api-access-ffxxk\") pod \"payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.629542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.629514    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8bafda-7d41-4661-800f-d2daf4f40a8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.629955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.628844    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-68f80c-dockercfg-r2kpz\""
Oct 30 22:32:07.630075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.628705    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-68f80c\""
Oct 30 22:32:07.687783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.687745    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"]
Oct 30 22:32:07.693516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.693490    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:07.711986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.711954    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"]
Oct 30 22:32:07.730937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.730881    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8bafda-7d41-4661-800f-d2daf4f40a8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.731125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.731108    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxxk\" (UniqueName: \"kubernetes.io/projected/fb8bafda-7d41-4661-800f-d2daf4f40a8b-kube-api-access-ffxxk\") pod \"payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.731614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.731597    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8bafda-7d41-4661-800f-d2daf4f40a8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.793948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.793874    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxxk\" (UniqueName: \"kubernetes.io/projected/fb8bafda-7d41-4661-800f-d2daf4f40a8b-kube-api-access-ffxxk\") pod \"payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.832024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.831992    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4cf\" (UniqueName: \"kubernetes.io/projected/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-kube-api-access-4f4cf\") pod \"payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:07.832227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.832132    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:07.832666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.832624    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:07.933831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.933794    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:07.934951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.934404    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4cf\" (UniqueName: \"kubernetes.io/projected/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-kube-api-access-4f4cf\") pod \"payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:07.934951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:07.934761    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:08.000099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:08.000067    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4cf\" (UniqueName: \"kubernetes.io/projected/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-kube-api-access-4f4cf\") pod \"payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:08.029615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:08.029588    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:08.209529 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:32:08.209487    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8bafda_7d41_4661_800f_d2daf4f40a8b.slice/crio-a81e17bddda6eba3f84aa3d56f942ea7cfa4567283f1ad15d8fe8628962124b9 WatchSource:0}: Error finding container a81e17bddda6eba3f84aa3d56f942ea7cfa4567283f1ad15d8fe8628962124b9: Status 404 returned error can't find the container with id a81e17bddda6eba3f84aa3d56f942ea7cfa4567283f1ad15d8fe8628962124b9
Oct 30 22:32:08.213603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:08.213581    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"]
Oct 30 22:32:08.284253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:08.284230    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" event={"ID":"fb8bafda-7d41-4661-800f-d2daf4f40a8b","Type":"ContainerStarted","Data":"a81e17bddda6eba3f84aa3d56f942ea7cfa4567283f1ad15d8fe8628962124b9"}
Oct 30 22:32:08.400616 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:32:08.400586    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a33ca6_1433_4f69_9617_b9e8c6ca65bf.slice/crio-14628a3ce2811ee4ce1c58998083613cb0f783668652bbfa03d176d6981e8802 WatchSource:0}: Error finding container 14628a3ce2811ee4ce1c58998083613cb0f783668652bbfa03d176d6981e8802: Status 404 returned error can't find the container with id 14628a3ce2811ee4ce1c58998083613cb0f783668652bbfa03d176d6981e8802
Oct 30 22:32:08.403767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:08.403744    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"]
Oct 30 22:32:09.295550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:09.295510    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d" event={"ID":"19a33ca6-1433-4f69-9617-b9e8c6ca65bf","Type":"ContainerStarted","Data":"143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f"}
Oct 30 22:32:09.296049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:09.295578    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:09.296049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:09.295604    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d" event={"ID":"19a33ca6-1433-4f69-9617-b9e8c6ca65bf","Type":"ContainerStarted","Data":"14628a3ce2811ee4ce1c58998083613cb0f783668652bbfa03d176d6981e8802"}
Oct 30 22:32:09.297227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:09.297204    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" event={"ID":"fb8bafda-7d41-4661-800f-d2daf4f40a8b","Type":"ContainerStarted","Data":"9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3"}
Oct 30 22:32:09.297347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:09.297329    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:09.413736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:09.413681    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d" podStartSLOduration=2.4136674510000002 podStartE2EDuration="2.413667451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:09.413107349 +0000 UTC m=+2705.747684118" watchObservedRunningTime="2026-10-30 22:32:09.413667451 +0000 UTC m=+2705.748244217"
Oct 30 22:32:09.538361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:09.538296    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" podStartSLOduration=2.538275514 podStartE2EDuration="2.538275514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:09.533747909 +0000 UTC m=+2705.868324680" watchObservedRunningTime="2026-10-30 22:32:09.538275514 +0000 UTC m=+2705.872852284"
Oct 30 22:32:11.543779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:11.543744    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:12.186282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:12.186247    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:12.618941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:12.618859    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:13.316363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:13.298657    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:13.700296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:13.700240    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:16.040160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:16.040121    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.006680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:17.006645    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.192911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:17.192877    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.385215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:17.385141    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.815752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:17.815701    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:17.988685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:17.987238    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:18.138932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:18.138832    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:18.263458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:18.263427    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"]
Oct 30 22:32:18.314157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:18.314129    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c301cf-6k96z"]
Oct 30 22:32:18.711976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:18.711946    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:19.789799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:19.789754    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204d1574-3f88-462a-8522-60af3d24338c" path="/var/lib/kubelet/pods/204d1574-3f88-462a-8522-60af3d24338c/volumes"
Oct 30 22:32:20.307012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:20.306980    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:32:20.307225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:20.307210    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:32:20.342401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:20.342371    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:20.534033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:20.533979    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:22.800661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:22.791151    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:24.050543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:24.050510    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"]
Oct 30 22:32:24.101336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:24.101302    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f2db6d-7xknd"]
Oct 30 22:32:25.093781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:25.093734    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:25.559090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:25.559050    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:25.709838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:25.709806    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:25.796455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:25.796419    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317e9b4e-16f5-43ef-96e5-2f5b427937c3" path="/var/lib/kubelet/pods/317e9b4e-16f5-43ef-96e5-2f5b427937c3/volumes"
Oct 30 22:32:25.907322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:25.907281    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:26.329496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:26.329387    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:27.099300 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:27.099261    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:27.420390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:27.420352    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:27.779468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:27.779389    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:32:27.988514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:27.988477    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:28.132075 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:32:28.131991    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:32:28.523671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:28.523054    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:32.111902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:32.111866    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:32.281924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:32.281887    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:32.439500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:32.439470    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:32.591812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:32.591784    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.053828 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:35.053792    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"]
Oct 30 22:32:35.070530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:35.070501    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d9e6a1a-sbfgm"]
Oct 30 22:32:35.365664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:35.365541    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.554551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:35.554513    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:35.789875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:35.789842    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993c87b6-f419-413c-9e41-f8585635cfee" path="/var/lib/kubelet/pods/993c87b6-f419-413c-9e41-f8585635cfee/volumes"
Oct 30 22:32:37.547223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:37.547192    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.272734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:38.272694    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:38.725042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:38.725004    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:39.237948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.237918    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"]
Oct 30 22:32:39.242029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.241996    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.300178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.300130    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-435923-openshift-default-dockercfg-txwm5\""
Oct 30 22:32:39.349264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.349157    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:39.403659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.402937    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"]
Oct 30 22:32:39.482664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.436539    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-socket\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.485946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.485908    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnsh\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-kube-api-access-djnsh\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.486251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.486232    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-credential-socket\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.492037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.491957    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-token\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.533789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.533751    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-envoy\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.534049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.534027    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-certs\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.534191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.534176    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-data\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.534317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.534301    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-podinfo\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.536353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.534493    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1891cfd-4b00-4edc-8303-226a3d4499fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.648179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.648129    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnsh\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-kube-api-access-djnsh\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.667798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.657526    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:39.668013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.667765    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-credential-socket\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.667906    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-token\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.667965    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-envoy\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.668014    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-certs\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.668074    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-data\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.668107    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-podinfo\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.668165    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-credential-socket\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.668223    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1891cfd-4b00-4edc-8303-226a3d4499fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.668339    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-socket\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.668776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.668751    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-socket\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.669236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.669198    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1891cfd-4b00-4edc-8303-226a3d4499fd-istiod-ca-cert\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.669473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.669452    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-certs\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.670029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.670005    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-data\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.671045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.671021    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-envoy\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.671282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.671260    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-podinfo\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.714185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.714136    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnsh\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-kube-api-access-djnsh\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.731319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.731295    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-token\") pod \"e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") " pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:39.861146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:39.861060    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:40.088033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.087995    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"]
Oct 30 22:32:40.091852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.091829    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5065faa-lt6v4"]
Oct 30 22:32:40.269228 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:32:40.269196    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1891cfd_4b00_4edc_8303_226a3d4499fd.slice/crio-b7f0338e48847029c39048b282a65747ebad0e79fdd3697efc760442529bc3f8 WatchSource:0}: Error finding container b7f0338e48847029c39048b282a65747ebad0e79fdd3697efc760442529bc3f8: Status 404 returned error can't find the container with id b7f0338e48847029c39048b282a65747ebad0e79fdd3697efc760442529bc3f8
Oct 30 22:32:40.272216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.272178    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:32:40.272299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.272254    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:32:40.272299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.272284    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:32:40.273257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.273233    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"]
Oct 30 22:32:40.672992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.672954    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" event={"ID":"a1891cfd-4b00-4edc-8303-226a3d4499fd","Type":"ContainerStarted","Data":"c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb"}
Oct 30 22:32:40.673172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.673020    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" event={"ID":"a1891cfd-4b00-4edc-8303-226a3d4499fd","Type":"ContainerStarted","Data":"b7f0338e48847029c39048b282a65747ebad0e79fdd3697efc760442529bc3f8"}
Oct 30 22:32:40.804043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.803973    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podStartSLOduration=1.803961123 podStartE2EDuration="1.803961123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:40.802813058 +0000 UTC m=+2737.137389829" watchObservedRunningTime="2026-10-30 22:32:40.803961123 +0000 UTC m=+2737.138537891"
Oct 30 22:32:40.861158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:40.861128    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:41.789372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:41.789334    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7676afea-3648-4d80-81c8-afbdb480adf7" path="/var/lib/kubelet/pods/7676afea-3648-4d80-81c8-afbdb480adf7/volumes"
Oct 30 22:32:41.867581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:41.867555    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:42.525730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:42.525692    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:42.686302 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:42.686270    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:42.687266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:42.687249    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:32:43.581967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:43.581930    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:43.735049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:43.735020    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:43.900287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:43.900238    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:46.816854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:46.816824    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:47.002044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:47.002015    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:47.543990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:47.543947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:51.052262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:51.052224    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"]
Oct 30 22:32:51.103040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:51.103007    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3a6ba9-t4c7q"]
Oct 30 22:32:51.789236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:51.789201    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8" path="/var/lib/kubelet/pods/e4f5dfcb-f6f1-4f5f-874a-a15df43fe8c8/volumes"
Oct 30 22:32:52.116114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:52.116026    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:52.912566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:52.912530    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:53.065592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:53.065556    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:56.828921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:32:56.828875    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:32:58.098911 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:32:58.098874    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:33:01.275798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:01.275760    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:01.431292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:01.431260    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:01.588193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:01.588111    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:05.195460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:05.195428    5503 scope.go:122] "RemoveContainer" containerID="f2e45e8fad1272e80bf654c643200ff08885634acba67616c65c5925db1386a4"
Oct 30 22:33:05.205093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:05.205073    5503 scope.go:122] "RemoveContainer" containerID="dbc1000a1bd7f31bd11a0259d035a0f782a91b1b66e3bf26a7928e67769fd85b"
Oct 30 22:33:05.213896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:05.213867    5503 scope.go:122] "RemoveContainer" containerID="89934caa9e0281c4df06ab28c1985b358000877029b1ffd040dd68c3a692f663"
Oct 30 22:33:06.134869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:06.134830    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:07.052378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:07.052339    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"]
Oct 30 22:33:07.102934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:07.102905    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a947af20-zbn4b"]
Oct 30 22:33:07.788779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:07.788745    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38420e1d-f42d-43a1-8161-d6c08c501b14" path="/var/lib/kubelet/pods/38420e1d-f42d-43a1-8161-d6c08c501b14/volumes"
Oct 30 22:33:08.051312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:08.051227    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"]
Oct 30 22:33:08.068910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:08.068881    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9331bb73-xfhq4"]
Oct 30 22:33:09.790198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:09.790164    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29cc565-15a9-468c-b564-72ff57db8089" path="/var/lib/kubelet/pods/a29cc565-15a9-468c-b564-72ff57db8089/volumes"
Oct 30 22:33:10.731350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:10.731317    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:15.285571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:15.285539    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:15.486167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:15.486133    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:15.542248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:15.542163    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:33:15.656149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:15.656109    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:19.051709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:19.051675    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"]
Oct 30 22:33:19.102124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:19.102097    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf03ea8-wqff4"]
Oct 30 22:33:19.789758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:19.789725    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5b032b-dd33-43f1-89ab-71f4949bccbe" path="/var/lib/kubelet/pods/7d5b032b-dd33-43f1-89ab-71f4949bccbe/volumes"
Oct 30 22:33:20.210051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.209993    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:20.531489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.531414    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"]
Oct 30 22:33:20.537194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.537177    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.591413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.591378    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"]
Oct 30 22:33:20.636913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.636885    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:33:20.637243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.637230    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:33:20.637288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.637244    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:33:20.641543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.641526    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:33:20.655408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.655379    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.655624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.655605    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8px\" (UniqueName: \"kubernetes.io/projected/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-kube-api-access-ql8px\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.655706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.655655    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.757050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.757018    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8px\" (UniqueName: \"kubernetes.io/projected/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-kube-api-access-ql8px\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.757179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.757069    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.757261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.757242    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.757862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.757841    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.759454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.759434    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.819542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.819468    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8px\" (UniqueName: \"kubernetes.io/projected/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-kube-api-access-ql8px\") pod \"maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:20.850434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:20.850407    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:21.221605 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:33:21.221566    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ea7862_5cb8_4b1d_9651_e877bf6cc8db.slice/crio-3da5de1dbcf038b89f29129f2edf789c851f50dad10e6617fd9b0e49eb596e91 WatchSource:0}: Error finding container 3da5de1dbcf038b89f29129f2edf789c851f50dad10e6617fd9b0e49eb596e91: Status 404 returned error can't find the container with id 3da5de1dbcf038b89f29129f2edf789c851f50dad10e6617fd9b0e49eb596e91
Oct 30 22:33:21.225226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:21.225196    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"]
Oct 30 22:33:21.956371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:21.956338    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks" event={"ID":"71ea7862-5cb8-4b1d-9651-e877bf6cc8db","Type":"ContainerStarted","Data":"50f4dc52a61de1d726f65d3ec83c23fa546d07f052459dea9ec7698829b0a395"}
Oct 30 22:33:21.956549 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:21.956414    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks" event={"ID":"71ea7862-5cb8-4b1d-9651-e877bf6cc8db","Type":"ContainerStarted","Data":"3da5de1dbcf038b89f29129f2edf789c851f50dad10e6617fd9b0e49eb596e91"}
Oct 30 22:33:22.077966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:22.077912    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks" podStartSLOduration=2.07789813 podStartE2EDuration="2.07789813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:22.076019861 +0000 UTC m=+2778.410596630" watchObservedRunningTime="2026-10-30 22:33:22.07789813 +0000 UTC m=+2778.412474899"
Oct 30 22:33:22.964854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:22.964820    5503 generic.go:356] "Generic (PLEG): container finished" podID="71ea7862-5cb8-4b1d-9651-e877bf6cc8db" containerID="50f4dc52a61de1d726f65d3ec83c23fa546d07f052459dea9ec7698829b0a395" exitCode=0
Oct 30 22:33:22.965250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:22.964859    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks" event={"ID":"71ea7862-5cb8-4b1d-9651-e877bf6cc8db","Type":"ContainerDied","Data":"50f4dc52a61de1d726f65d3ec83c23fa546d07f052459dea9ec7698829b0a395"}
Oct 30 22:33:24.119974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.119948    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:24.195380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.195348    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-openshift-ca\") pod \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") "
Oct 30 22:33:24.195591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.195567    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-xks-ca\") pod \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") "
Oct 30 22:33:24.195728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.195709    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-kube-api-access-ql8px\" (UniqueName: \"kubernetes.io/projected/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-kube-api-access-ql8px\") pod \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\" (UID: \"71ea7862-5cb8-4b1d-9651-e877bf6cc8db\") "
Oct 30 22:33:24.195807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.195787    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-openshift-ca" pod "71ea7862-5cb8-4b1d-9651-e877bf6cc8db" (UID: "71ea7862-5cb8-4b1d-9651-e877bf6cc8db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:24.196207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.196184    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.197736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.197710    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-xks-ca" pod "71ea7862-5cb8-4b1d-9651-e877bf6cc8db" (UID: "71ea7862-5cb8-4b1d-9651-e877bf6cc8db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:24.197826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.197732    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-kube-api-access-ql8px" pod "71ea7862-5cb8-4b1d-9651-e877bf6cc8db" (UID: "71ea7862-5cb8-4b1d-9651-e877bf6cc8db"). InnerVolumeSpecName "kube-api-access-ql8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:24.297657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.297572    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.297657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.297601    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8px\" (UniqueName: \"kubernetes.io/projected/71ea7862-5cb8-4b1d-9651-e877bf6cc8db-kube-api-access-ql8px\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.979722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.979685    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks" event={"ID":"71ea7862-5cb8-4b1d-9651-e877bf6cc8db","Type":"ContainerDied","Data":"3da5de1dbcf038b89f29129f2edf789c851f50dad10e6617fd9b0e49eb596e91"}
Oct 30 22:33:24.979722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.979708    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"
Oct 30 22:33:24.979990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:24.979732    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3da5de1dbcf038b89f29129f2edf789c851f50dad10e6617fd9b0e49eb596e91"
Oct 30 22:33:25.075346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:25.075317    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:25.451254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:25.451225    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:26.667988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:26.667949    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:28.099390 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:33:28.099354    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:33:29.977676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:29.977622    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:30.053877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:30.053839    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"]
Oct 30 22:33:30.102426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:30.102396    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4d2f6b1-l88zn"]
Oct 30 22:33:31.788980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:31.788950    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a18050e-6f0e-4504-8fc3-6cf63e0f89f9" path="/var/lib/kubelet/pods/5a18050e-6f0e-4504-8fc3-6cf63e0f89f9/volumes"
Oct 30 22:33:32.460001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:32.459967    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:32.623523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:32.623478    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:34.514931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:34.514903    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:34.674139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:34.674089    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:34.825762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:34.825672    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:39.158716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:39.158684    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:41.058460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:41.058426    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"]
Oct 30 22:33:41.109176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:41.109139    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d74c19-2wvrq"]
Oct 30 22:33:41.788748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:41.788714    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de45cc76-477a-4813-af22-3007a3748e10" path="/var/lib/kubelet/pods/de45cc76-477a-4813-af22-3007a3748e10/volumes"
Oct 30 22:33:43.732924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:43.732885    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.051233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:48.051163    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"]
Oct 30 22:33:48.054170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:48.054147    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc553e2d-k5pgs"]
Oct 30 22:33:48.100898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:48.100869    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:33:48.475757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:48.475695    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.643076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:48.643039    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:48.805709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:48.805062    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:49.788828 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:49.788793    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d20c3c-40e5-4736-8ba9-4d90e988395a" path="/var/lib/kubelet/pods/d6d20c3c-40e5-4736-8ba9-4d90e988395a/volumes"
Oct 30 22:33:53.417660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:53.417602    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:33:58.098401 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:33:58.098363    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:33:58.124356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:33:58.124328    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:02.812645 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:02.812596    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:05.248454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:05.248420    5503 scope.go:122] "RemoveContainer" containerID="3f93db148363992da06608bd6144deaaacba213f4b3fd7e34d428643ec6d82b2"
Oct 30 22:34:05.258432 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:05.258410    5503 scope.go:122] "RemoveContainer" containerID="0670c0e9abbe38b1ae4c9a1e64cb70715593623ff794b36c2685be5ad4fc4746"
Oct 30 22:34:05.269883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:05.269865    5503 scope.go:122] "RemoveContainer" containerID="580ff4cd10fa475089684e7eceee4d58bdf5e2b6818a6bcc2a069f4c1049a0a1"
Oct 30 22:34:05.280162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:05.280146    5503 scope.go:122] "RemoveContainer" containerID="a68f0a4bf85505642009339f65ceec934940be64161e5a9db13ffb6c05bffe17"
Oct 30 22:34:05.288792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:05.288775    5503 scope.go:122] "RemoveContainer" containerID="e3a82f8e39a77e75f7bac45b92a38b30f4502dc3d1f8f834ff149e0fa7642206"
Oct 30 22:34:05.296893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:05.296875    5503 scope.go:122] "RemoveContainer" containerID="697f743f4ab75d9b332d1088297247848cbd52b47db2eeaf1685c359208f186f"
Oct 30 22:34:05.305082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:05.305055    5503 scope.go:122] "RemoveContainer" containerID="ab10c15362ff88fafa3d3938650cf9f7d304f373da11ee61ed3582489ea4684c"
Oct 30 22:34:07.051566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:07.051527    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"]
Oct 30 22:34:07.102213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:07.102182    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92ac6806-r4xwk"]
Oct 30 22:34:07.514089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:07.514039    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:07.788975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:07.788895    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682118b1-1dbb-4422-9ad3-7af795df22d9" path="/var/lib/kubelet/pods/682118b1-1dbb-4422-9ad3-7af795df22d9/volumes"
Oct 30 22:34:12.010118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:12.010085    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:16.618334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:16.618299    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:19.959506 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:19.959473    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:34:21.053625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:21.053587    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"]
Oct 30 22:34:21.084643 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:21.084599    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a55a66-gl96s"]
Oct 30 22:34:21.627307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:21.627248    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:21.788612 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:21.788574    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c724cf-9ad6-4774-9054-6bc794a174ce" path="/var/lib/kubelet/pods/77c724cf-9ad6-4774-9054-6bc794a174ce/volumes"
Oct 30 22:34:22.053394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:22.053357    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"]
Oct 30 22:34:22.104056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:22.104021    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc3b91a-znzzf"]
Oct 30 22:34:22.181494 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:22.181463    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:23.051088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:23.051054    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"]
Oct 30 22:34:23.101242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:23.101208    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a68cbd2-4pb7f"]
Oct 30 22:34:23.790956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:23.790925    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b55996-e3a2-4367-a9cc-5d2e5c90e324" path="/var/lib/kubelet/pods/09b55996-e3a2-4367-a9cc-5d2e5c90e324/volumes"
Oct 30 22:34:23.791391 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:23.791364    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14dc26a-269c-4bdb-aaeb-8546c5a460e7" path="/var/lib/kubelet/pods/f14dc26a-269c-4bdb-aaeb-8546c5a460e7/volumes"
Oct 30 22:34:25.788752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:25.788716    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:26.223244 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:26.223205    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:27.018255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:27.018209    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:27.193872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:27.192656    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:28.100033 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:34:28.099994    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:34:28.727812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:28.727770    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:34:28.728008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:28.727821    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:34:30.307570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.307536    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"]
Oct 30 22:34:30.309852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.308176    5503 cpu_manager.go:414] "containerMap: removing container" podUID="71ea7862-5cb8-4b1d-9651-e877bf6cc8db" containerName="revoke-keys"
Oct 30 22:34:30.309852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.308192    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ea7862-5cb8-4b1d-9651-e877bf6cc8db" containerName="revoke-keys"
Oct 30 22:34:30.309852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.308354    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ea7862-5cb8-4b1d-9651-e877bf6cc8db" containerName="revoke-keys"
Oct 30 22:34:30.311513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.311494    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.368887 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.368861    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"]
Oct 30 22:34:30.414788 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.414764    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:34:30.414913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.414828    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:34:30.414913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.414838    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:34:30.415772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.415756    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:34:30.452209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.452181    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.452325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.452214    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.452389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.452354    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w922\" (UniqueName: \"kubernetes.io/projected/630cbefe-5690-49e4-8450-424c84c50d2c-kube-api-access-4w922\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.458691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.458667    5503 generic.go:356] "Generic (PLEG): container finished" podID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerID="c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb" exitCode=0
Oct 30 22:34:30.458787 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.458736    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" event={"ID":"a1891cfd-4b00-4edc-8303-226a3d4499fd","Type":"ContainerDied","Data":"c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb"}
Oct 30 22:34:30.459110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.459097    5503 scope.go:122] "RemoveContainer" containerID="c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb"
Oct 30 22:34:30.460496 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.460463    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:34:30.460551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.460535    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:34:30.460594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.460577    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:34:30.553486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.553461    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.553660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.553502    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.553740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.553716    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w922\" (UniqueName: \"kubernetes.io/projected/630cbefe-5690-49e4-8450-424c84c50d2c-kube-api-access-4w922\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.554317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.554292    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.555794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.555761    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.623467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.623383    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w922\" (UniqueName: \"kubernetes.io/projected/630cbefe-5690-49e4-8450-424c84c50d2c-kube-api-access-4w922\") pod \"maas-api-revoke-sub-3b6e88e7-spkvq\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:30.626997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:30.626974    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:31.014946 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:34:31.014917    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630cbefe_5690_49e4_8450_424c84c50d2c.slice/crio-dcb92440868b4a144b34f3f2bdd4ecbfc32eb74a22129154204d8165aeb9d0cd WatchSource:0}: Error finding container dcb92440868b4a144b34f3f2bdd4ecbfc32eb74a22129154204d8165aeb9d0cd: Status 404 returned error can't find the container with id dcb92440868b4a144b34f3f2bdd4ecbfc32eb74a22129154204d8165aeb9d0cd
Oct 30 22:34:31.017480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.017375    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"]
Oct 30 22:34:31.035185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.035064    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:31.467192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.467146    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq" event={"ID":"630cbefe-5690-49e4-8450-424c84c50d2c","Type":"ContainerStarted","Data":"2d078af5a972a0a11593d215d75cb2f53b127074c4465486ce91e40b6aa7ba6e"}
Oct 30 22:34:31.467631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.467249    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq" event={"ID":"630cbefe-5690-49e4-8450-424c84c50d2c","Type":"ContainerStarted","Data":"dcb92440868b4a144b34f3f2bdd4ecbfc32eb74a22129154204d8165aeb9d0cd"}
Oct 30 22:34:31.469236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.469214    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" event={"ID":"a1891cfd-4b00-4edc-8303-226a3d4499fd","Type":"ContainerStarted","Data":"aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339"}
Oct 30 22:34:31.596975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.596920    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq" podStartSLOduration=1.5969062950000001 podStartE2EDuration="1.596906295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:31.594555026 +0000 UTC m=+2847.929131790" watchObservedRunningTime="2026-10-30 22:34:31.596906295 +0000 UTC m=+2847.931483063"
Oct 30 22:34:31.862038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.862005    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:34:31.862217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.862198    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:34:31.862285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:31.862261    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 22:34:32.481161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:32.479851    5503 generic.go:356] "Generic (PLEG): container finished" podID="630cbefe-5690-49e4-8450-424c84c50d2c" containerID="2d078af5a972a0a11593d215d75cb2f53b127074c4465486ce91e40b6aa7ba6e" exitCode=0
Oct 30 22:34:32.481161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:32.481100    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq" event={"ID":"630cbefe-5690-49e4-8450-424c84c50d2c","Type":"ContainerDied","Data":"2d078af5a972a0a11593d215d75cb2f53b127074c4465486ce91e40b6aa7ba6e"}
Oct 30 22:34:32.862379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:32.862298    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:34:32.862379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:32.862366    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 22:34:33.675539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.675512    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:33.801488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.801411    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-xks-ca\") pod \"630cbefe-5690-49e4-8450-424c84c50d2c\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") "
Oct 30 22:34:33.801690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.801540    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-openshift-ca\") pod \"630cbefe-5690-49e4-8450-424c84c50d2c\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") "
Oct 30 22:34:33.801690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.801586    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630cbefe-5690-49e4-8450-424c84c50d2c-kube-api-access-4w922\" (UniqueName: \"kubernetes.io/projected/630cbefe-5690-49e4-8450-424c84c50d2c-kube-api-access-4w922\") pod \"630cbefe-5690-49e4-8450-424c84c50d2c\" (UID: \"630cbefe-5690-49e4-8450-424c84c50d2c\") "
Oct 30 22:34:33.801935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.801909    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-openshift-ca" pod "630cbefe-5690-49e4-8450-424c84c50d2c" (UID: "630cbefe-5690-49e4-8450-424c84c50d2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:33.803899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.803873    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-xks-ca" pod "630cbefe-5690-49e4-8450-424c84c50d2c" (UID: "630cbefe-5690-49e4-8450-424c84c50d2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:33.804504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.804479    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630cbefe-5690-49e4-8450-424c84c50d2c-kube-api-access-4w922" pod "630cbefe-5690-49e4-8450-424c84c50d2c" (UID: "630cbefe-5690-49e4-8450-424c84c50d2c"). InnerVolumeSpecName "kube-api-access-4w922". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:33.861414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.861376    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:34:33.861587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.861438    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 22:34:33.903068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.903035    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:33.903068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.903069    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/630cbefe-5690-49e4-8450-424c84c50d2c-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:33.903332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:33.903088    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w922\" (UniqueName: \"kubernetes.io/projected/630cbefe-5690-49e4-8450-424c84c50d2c-kube-api-access-4w922\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:34.494519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:34.494484    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq" event={"ID":"630cbefe-5690-49e4-8450-424c84c50d2c","Type":"ContainerDied","Data":"dcb92440868b4a144b34f3f2bdd4ecbfc32eb74a22129154204d8165aeb9d0cd"}
Oct 30 22:34:34.494735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:34.494537    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb92440868b4a144b34f3f2bdd4ecbfc32eb74a22129154204d8165aeb9d0cd"
Oct 30 22:34:34.494735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:34.494498    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"
Oct 30 22:34:34.862219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:34.861772    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:34:34.862872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:34.862832    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 22:34:35.300215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:35.300185    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:35.861762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:35.861727    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body=
Oct 30 22:34:35.861917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:35.861788    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused"
Oct 30 22:34:36.101143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:36.101109    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:36.866334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:36.866304    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:34:36.866686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:36.866665    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:34:36.867405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:36.867388    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:34:37.097071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:37.097038    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"]
Oct 30 22:34:37.099110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:37.099085    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cb0cb8b-dhg5h"]
Oct 30 22:34:37.789608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:37.789571    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9bf3be-be04-4b49-b8fe-ba65430e9925" path="/var/lib/kubelet/pods/4c9bf3be-be04-4b49-b8fe-ba65430e9925/volumes"
Oct 30 22:34:40.924238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:40.924200    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:44.052151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:44.052116    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"]
Oct 30 22:34:44.102842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:44.102814    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29ba184-m8pls"]
Oct 30 22:34:45.050480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:45.050438    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"]
Oct 30 22:34:45.101699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:45.101665    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1024e4c7-8qm7p"]
Oct 30 22:34:45.620221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:45.620189    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:45.789423 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:45.789388    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00b21a54-db8e-452e-a44e-2d1788e48b9a" path="/var/lib/kubelet/pods/00b21a54-db8e-452e-a44e-2d1788e48b9a/volumes"
Oct 30 22:34:45.789854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:45.789839    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88414f5d-2368-4704-b519-8d5c46238481" path="/var/lib/kubelet/pods/88414f5d-2368-4704-b519-8d5c46238481/volumes"
Oct 30 22:34:46.058826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:46.058793    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:46.213124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:46.213083    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:50.345789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:50.345754    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:50.528715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:50.528680    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:51.027217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.027181    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:51.656690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.651490    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"]
Oct 30 22:34:51.656690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.652567    5503 cpu_manager.go:414] "containerMap: removing container" podUID="630cbefe-5690-49e4-8450-424c84c50d2c" containerName="revoke-keys"
Oct 30 22:34:51.656690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.652602    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630cbefe-5690-49e4-8450-424c84c50d2c" containerName="revoke-keys"
Oct 30 22:34:51.656690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.652860    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="630cbefe-5690-49e4-8450-424c84c50d2c" containerName="revoke-keys"
Oct 30 22:34:51.784684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.780000    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:51.784684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.780804    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"]
Oct 30 22:34:51.784684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.780959    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.838319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.838288    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-74bb81-openshift-default-dockercfg-fh8pl\""
Oct 30 22:34:51.895667 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.882397    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4j6\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-kube-api-access-fs4j6\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.896131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.896088    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-token\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.896345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.896329    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-podinfo\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.896483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.896465    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istiod-ca-cert\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.896662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.896625    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-credential-socket\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.896929 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.896906    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-socket\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.905682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.897173    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-certs\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.906035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.906000    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-data\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:51.906246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:51.906222    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-envoy\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.012660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.012538    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-certs\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.013426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.013399    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-data\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.014036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.014014    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-envoy\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.014287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.014270    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4j6\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-kube-api-access-fs4j6\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.014421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.014407    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-token\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.014540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.014526    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-podinfo\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.014716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.014698    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istiod-ca-cert\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.014857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.014841    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-credential-socket\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.015027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.015013    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-socket\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.022334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.015563    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-socket\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.022965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.013296    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-certs\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.023159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.013934    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-data\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.023671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.023605    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:52.024745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.024722    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-credential-socket\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.024910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.024890    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istiod-ca-cert\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.031547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.031516    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-podinfo\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.031964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.031904    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-envoy\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.081427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.081399    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4j6\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-kube-api-access-fs4j6\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.085884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.085860    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-token\") pod \"e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") " pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.104101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.104073    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:52.530095 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:34:52.530063    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f7f72d_4152_4235_a2e2_ab6404fbcf33.slice/crio-e2ad74f779f65ff16154a22e4b39c9818b5d8576651ea2d9b0bfa1fbb20b89f9 WatchSource:0}: Error finding container e2ad74f779f65ff16154a22e4b39c9818b5d8576651ea2d9b0bfa1fbb20b89f9: Status 404 returned error can't find the container with id e2ad74f779f65ff16154a22e4b39c9818b5d8576651ea2d9b0bfa1fbb20b89f9
Oct 30 22:34:52.532993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.532951    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:34:52.533094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.533042    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:34:52.533094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.533087    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:34:52.533479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.533458    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"]
Oct 30 22:34:52.634218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.634180    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" event={"ID":"e0f7f72d-4152-4235-a2e2-ab6404fbcf33","Type":"ContainerStarted","Data":"465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894"}
Oct 30 22:34:52.634393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.634247    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" event={"ID":"e0f7f72d-4152-4235-a2e2-ab6404fbcf33","Type":"ContainerStarted","Data":"e2ad74f779f65ff16154a22e4b39c9818b5d8576651ea2d9b0bfa1fbb20b89f9"}
Oct 30 22:34:52.760648 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:52.760594    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" podStartSLOduration=1.760579147 podStartE2EDuration="1.760579147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:52.758993403 +0000 UTC m=+2869.093570173" watchObservedRunningTime="2026-10-30 22:34:52.760579147 +0000 UTC m=+2869.095155916"
Oct 30 22:34:53.104429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:53.104399    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:53.109574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:53.109548    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:53.640608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:53.640581    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:53.641904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:53.641880    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:34:54.978491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:54.978455    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:56.285001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:56.284966    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:56.496671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:56.496627    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:56.649196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:56.649111    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:58.099190 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:34:58.099152    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:34:59.599690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:59.599631    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:34:59.759032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:34:59.753815    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:00.052376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:00.052344    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"]
Oct 30 22:35:00.102743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:00.102711    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"]
Oct 30 22:35:00.111398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:00.111361    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392eea77-vljvv"]
Oct 30 22:35:00.111724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:00.111706    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0a62a8-mz72z"]
Oct 30 22:35:00.183594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:00.183560    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:01.788533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:01.788502    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3" path="/var/lib/kubelet/pods/0ab2066c-8fb3-40a5-81eb-2fc9f27ca4d3/volumes"
Oct 30 22:35:01.788997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:01.788981    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35726435-9e69-4cf9-87ff-c0c4cb7c6962" path="/var/lib/kubelet/pods/35726435-9e69-4cf9-87ff-c0c4cb7c6962/volumes"
Oct 30 22:35:01.843816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:01.843780    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:05.126441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:05.126404    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:05.355897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:05.355866    5503 scope.go:122] "RemoveContainer" containerID="cc4cc92a2fcefae8b1a1687bcc496a1924e4af084dc3a623c4b0194a4c3cf928"
Oct 30 22:35:05.365780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:05.365755    5503 scope.go:122] "RemoveContainer" containerID="a4530251cbfd51f20ac7ebc2f30a12f672eed6f82037f240aa19b29f0c21d7ff"
Oct 30 22:35:05.375180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:05.375159    5503 scope.go:122] "RemoveContainer" containerID="0c1ed4d8c72871427675e334b49ae759545c9b50cc414989a79cd61030b6bdca"
Oct 30 22:35:05.383419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:05.383402    5503 scope.go:122] "RemoveContainer" containerID="d1855e6e670b933a88524ad28ae7f1b2e4d14cbe4345b4e464e0e05254396a4d"
Oct 30 22:35:05.392348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:05.392324    5503 scope.go:122] "RemoveContainer" containerID="0320abbd4e2d339a29dd9fd6ec1a4f41f064263409916d33738cbc19fc158403"
Oct 30 22:35:05.401229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:05.401204    5503 scope.go:122] "RemoveContainer" containerID="1c4cb89d0c4def6633897cd66413d6292ec5ceb949bb91d0383b017206ce92af"
Oct 30 22:35:09.293692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:09.293654    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:35:09.638568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:09.638484    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:12.051193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:12.051159    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"]
Oct 30 22:35:12.102017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:12.101987    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0fa2f2-7t799"]
Oct 30 22:35:13.788753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:13.788680    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd82aa66-e922-478f-bdcb-a75b3bb5ef86" path="/var/lib/kubelet/pods/fd82aa66-e922-478f-bdcb-a75b3bb5ef86/volumes"
Oct 30 22:35:14.218161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:14.218124    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:18.721602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:18.721567    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:23.243800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:23.243766    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:26.113818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:26.113766    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:27.333822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:27.333786    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:27.881984 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:27.881931    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:28.105093 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:35:28.105056    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:35:29.100285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:29.100254    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"]
Oct 30 22:35:29.102605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:29.102582    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8c1677-s796n"]
Oct 30 22:35:29.413201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:29.413151    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:29.609850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:29.609811    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:29.790965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:29.790891    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b114d3c-7249-4ec6-9ae9-36e8d1a1e735" path="/var/lib/kubelet/pods/2b114d3c-7249-4ec6-9ae9-36e8d1a1e735/volumes"
Oct 30 22:35:32.440976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:32.440942    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:35.941238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:35.941184    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:36.384822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:36.384714    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:37.906592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:37.906536    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:38.119571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:38.119539    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:38.269208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:38.269121    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:38.375973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:38.375934    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"]
Oct 30 22:35:38.426855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:38.426817    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1875b4-sfpks"]
Oct 30 22:35:38.835479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:38.835442    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:39.789786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:39.789756    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ea7862-5cb8-4b1d-9651-e877bf6cc8db" path="/var/lib/kubelet/pods/71ea7862-5cb8-4b1d-9651-e877bf6cc8db/volumes"
Oct 30 22:35:42.958202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:42.958162    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:43.895426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:43.895389    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:45.500994 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.500962    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"]
Oct 30 22:35:45.505032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.505011    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.560626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.560596    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"]
Oct 30 22:35:45.581570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.581544    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bf1218-0919-4fda-80b5-c206c80bc169-tls-certs\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.581736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.581704    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-dshm\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.581781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.581749    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-tmp-dir\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.581781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.581772    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hss9g\" (UniqueName: \"kubernetes.io/projected/f9bf1218-0919-4fda-80b5-c206c80bc169-kube-api-access-hss9g\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.581946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.581933    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-home\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.581983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.581961    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-model-cache\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.608418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.608390    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc6466\"/\"kube-root-ca.crt\""
Oct 30 22:35:45.608668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.608528    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc6466\"/\"auth-test-model-fc6466-kserve-self-signed-certs\""
Oct 30 22:35:45.608782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.608735    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:45.609849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.609830    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc6466\"/\"openshift-service-ca.crt\""
Oct 30 22:35:45.609981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.609964    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc6466\"/\"default-dockercfg-78lvb\""
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.688991    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hss9g\" (UniqueName: \"kubernetes.io/projected/f9bf1218-0919-4fda-80b5-c206c80bc169-kube-api-access-hss9g\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.689156    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-home\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.689208    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-model-cache\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.689269    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bf1218-0919-4fda-80b5-c206c80bc169-tls-certs\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.689439    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-dshm\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.689503    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-tmp-dir\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.690471    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-tmp-dir\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.691109    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-home\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.692659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.691365    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-model-cache\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.696573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.696544    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bf1218-0919-4fda-80b5-c206c80bc169-tls-certs\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.701792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.700373    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-dshm\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.753556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.753486    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hss9g\" (UniqueName: \"kubernetes.io/projected/f9bf1218-0919-4fda-80b5-c206c80bc169-kube-api-access-hss9g\") pod \"auth-test-model-fc6466-kserve-76bb69cd97-l66qv\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") " pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:45.817303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:45.817271    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:46.207989 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:35:46.207949    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9bf1218_0919_4fda_80b5_c206c80bc169.slice/crio-aa4442ac79231e62b8e663cca4f8cd86e621d1be10f31bf42b114bd49f252cf4 WatchSource:0}: Error finding container aa4442ac79231e62b8e663cca4f8cd86e621d1be10f31bf42b114bd49f252cf4: Status 404 returned error can't find the container with id aa4442ac79231e62b8e663cca4f8cd86e621d1be10f31bf42b114bd49f252cf4
Oct 30 22:35:46.210258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:46.210233    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"]
Oct 30 22:35:47.090691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:47.090630    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv" event={"ID":"f9bf1218-0919-4fda-80b5-c206c80bc169","Type":"ContainerStarted","Data":"04af6fad708c97226087ce59a76af71cdfaed8c2b02eab9de6a471889754457b"}
Oct 30 22:35:47.090691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:47.090695    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv" event={"ID":"f9bf1218-0919-4fda-80b5-c206c80bc169","Type":"ContainerStarted","Data":"aa4442ac79231e62b8e663cca4f8cd86e621d1be10f31bf42b114bd49f252cf4"}
Oct 30 22:35:47.216045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:47.215988    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv" podStartSLOduration=2.215970849 podStartE2EDuration="2.215970849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:47.212470947 +0000 UTC m=+2923.547047716" watchObservedRunningTime="2026-10-30 22:35:47.215970849 +0000 UTC m=+2923.550547617"
Oct 30 22:35:47.491027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:47.490991    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:47.818017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:47.817937    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:52.015806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:52.015765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:55.817752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:55.817716    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:55.830041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:55.830017    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:56.160443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:56.160396    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:35:56.529928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:35:56.529845    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:35:58.099701 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:35:58.099660    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:36:00.317561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:00.317530    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:36:01.094455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:01.094420    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.253032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:01.252976    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:01.845902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:01.845864    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:05.450344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.450302    5503 scope.go:122] "RemoveContainer" containerID="a9820c397bb31b58cf54802e1fd0d6a8d8a4cbf8dfe27fb88b949d3c17820798"
Oct 30 22:36:05.460045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.460020    5503 scope.go:122] "RemoveContainer" containerID="8be6ff6979c3b6da0c0a1cb3f4655321256e7f02834aaa8cbb747b9334b9b2d8"
Oct 30 22:36:05.468700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.468680    5503 scope.go:122] "RemoveContainer" containerID="3d7c6a7b4ed540db49034d64b6acf91fbc75dfbfd50cc24b772142cb86b557af"
Oct 30 22:36:05.477005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.476986    5503 scope.go:122] "RemoveContainer" containerID="469e0248a81c7d7906230bb30c1025bc0986bd17d17765ac86faea15d9d2c2fb"
Oct 30 22:36:05.485162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.485145    5503 scope.go:122] "RemoveContainer" containerID="ed230ed7765e947cd935210314bd68fba8773c53dd9d0435f2b2ca9c9436e82f"
Oct 30 22:36:05.492924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.492902    5503 scope.go:122] "RemoveContainer" containerID="25da2987857f700f7e1c83d1512ce034e98c291459df5d3ff00498316e799168"
Oct 30 22:36:05.501296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.501280    5503 scope.go:122] "RemoveContainer" containerID="1098db3fcd1e147350df394b71a1784b9ff824c644461ee1a083474a5fc7b8ca"
Oct 30 22:36:05.509782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.509765    5503 scope.go:122] "RemoveContainer" containerID="a9c12809d31d2fc252f0b44ca87dce2bb3bb813c21c5b5b127e0ef3f38bc8906"
Oct 30 22:36:05.946297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:05.946247    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:10.618863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:10.618829    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:15.281021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:15.280944    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:19.252831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:19.252763    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:19.772804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:19.772765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:20.626781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:20.626747    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"]
Oct 30 22:36:20.627248 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:20.626957    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" containerID="cri-o://3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a" gracePeriod=30
Oct 30 22:36:20.677375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:20.677341    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"]
Oct 30 22:36:20.677551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:20.677528    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" containerID="cri-o://c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a" gracePeriod=30
Oct 30 22:36:21.028683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:21.028619    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:21.690090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:21.690044    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:24.376921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:24.376890    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:25.820437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:25.820400    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Oct 30 22:36:25.820905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:25.820465    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Oct 30 22:36:25.947743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:25.947719    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:36:25.986056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:25.986031    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:36:26.066436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066400    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-credential-socket\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066495    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-envoy\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066558    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-data\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066697    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-socket\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066739    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-token\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-token\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066778    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-podinfo\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066786    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-credential-socket" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.066952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066812    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ec8cd4-71b2-4643-a622-f557a2728ca6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ec8cd4-71b2-4643-a622-f557a2728ca6-istiod-ca-cert\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066862    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-kube-api-access-pbrrh\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-kube-api-access-pbrrh\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066912    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-certs\") pod \"15ec8cd4-71b2-4643-a622-f557a2728ca6\" (UID: \"15ec8cd4-71b2-4643-a622-f557a2728ca6\") "
Oct 30 22:36:26.066952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.066940    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-data" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.067897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.067230    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ec8cd4-71b2-4643-a622-f557a2728ca6-istiod-ca-cert" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:26.067897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.067458    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-certs" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.067897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.067548    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-socket" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.067897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.067597    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.067897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.067620    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.067897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.067658    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ec8cd4-71b2-4643-a622-f557a2728ca6-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.067897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.067676    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.069379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.069351    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-envoy" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.069530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.069408    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-podinfo" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:26.069656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.069622    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-token" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:26.069908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.069887    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-kube-api-access-pbrrh" pod "15ec8cd4-71b2-4643-a622-f557a2728ca6" (UID: "15ec8cd4-71b2-4643-a622-f557a2728ca6"). InnerVolumeSpecName "kube-api-access-pbrrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:26.168597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.168569    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-kube-api-access-t9zks\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-kube-api-access-t9zks\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.168790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.168610    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istiod-ca-cert\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.168790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.168707    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-envoy\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.168790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.168741    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-token\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-token\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.168949 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.168827    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-socket\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.168949 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.168881    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-data\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.169068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.168945    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-credential-socket\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.169068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169013    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-podinfo\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.169184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169057    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istiod-ca-cert" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:26.169184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169075    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-certs\") pod \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\" (UID: \"131bcd3e-372f-47ad-aca8-72a0a92d7d53\") "
Oct 30 22:36:26.169184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169164    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-socket" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.169379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169359    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-credential-socket" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.169440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169374    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-data" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.169601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169578    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-certs" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.169774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169752    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.169845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169780    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.169845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169796    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.169845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169812    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.169845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169828    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.170085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169843    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrrh\" (UniqueName: \"kubernetes.io/projected/15ec8cd4-71b2-4643-a622-f557a2728ca6-kube-api-access-pbrrh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.170085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169859    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.170085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169874    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.170085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169888    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ec8cd4-71b2-4643-a622-f557a2728ca6-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.170085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.169903    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.171056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.171022    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-envoy" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:26.171256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.171237    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-token" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:26.171348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.171331    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-kube-api-access-t9zks" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "kube-api-access-t9zks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:26.171550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.171515    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-podinfo" pod "131bcd3e-372f-47ad-aca8-72a0a92d7d53" (UID: "131bcd3e-372f-47ad-aca8-72a0a92d7d53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:26.271414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.271375    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.271414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.271414    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zks\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-kube-api-access-t9zks\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.271659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.271428    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.271659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.271442    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131bcd3e-372f-47ad-aca8-72a0a92d7d53-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:26.382348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.382313    5503 generic.go:356] "Generic (PLEG): container finished" podID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerID="3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a" exitCode=0
Oct 30 22:36:26.382547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.382370    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"
Oct 30 22:36:26.382547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.382393    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" event={"ID":"15ec8cd4-71b2-4643-a622-f557a2728ca6","Type":"ContainerDied","Data":"3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a"}
Oct 30 22:36:26.382547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.382453    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s" event={"ID":"15ec8cd4-71b2-4643-a622-f557a2728ca6","Type":"ContainerDied","Data":"631289275d4b29e67a7dfea5bfcd46cfbcf0ee7299d6718b5c8a676ef0eff39a"}
Oct 30 22:36:26.382547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.382481    5503 scope.go:122] "RemoveContainer" containerID="3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a"
Oct 30 22:36:26.383795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.383772    5503 generic.go:356] "Generic (PLEG): container finished" podID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerID="c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a" exitCode=0
Oct 30 22:36:26.383920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.383810    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"
Oct 30 22:36:26.383920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.383857    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" event={"ID":"131bcd3e-372f-47ad-aca8-72a0a92d7d53","Type":"ContainerDied","Data":"c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a"}
Oct 30 22:36:26.383920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.383908    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" event={"ID":"131bcd3e-372f-47ad-aca8-72a0a92d7d53","Type":"ContainerDied","Data":"3d9aba922626dde5af2641529b32b527c4272c8349f4466d4f6c1ff4f5d5d74d"}
Oct 30 22:36:26.394129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.394109    5503 scope.go:122] "RemoveContainer" containerID="3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a"
Oct 30 22:36:26.394490 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:36:26.394471    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a\": container with ID starting with 3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a not found: ID does not exist" containerID="3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a"
Oct 30 22:36:26.394554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.394498    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a"} err="failed to get container status \"3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a\": rpc error: code = NotFound desc = could not find container \"3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a\": container with ID starting with 3af8251a62e815d4f1ba0b5ad5436188ad697808358d126bb7e0f92470ebd21a not found: ID does not exist"
Oct 30 22:36:26.394554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.394513    5503 scope.go:122] "RemoveContainer" containerID="c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a"
Oct 30 22:36:26.403924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.403906    5503 scope.go:122] "RemoveContainer" containerID="c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a"
Oct 30 22:36:26.404179 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:36:26.404163    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a\": container with ID starting with c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a not found: ID does not exist" containerID="c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a"
Oct 30 22:36:26.404228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.404185    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a"} err="failed to get container status \"c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a\": rpc error: code = NotFound desc = could not find container \"c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a\": container with ID starting with c1104a693fb1eb6e26e12a73d21e50522cfa4b213c244bd11b6acee5a2b8a94a not found: ID does not exist"
Oct 30 22:36:26.450278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.450220    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:26.553802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.553770    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"]
Oct 30 22:36:26.558449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.558424    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-qgz9s"]
Oct 30 22:36:26.673963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.673934    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"]
Oct 30 22:36:26.678081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.678058    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5"]
Oct 30 22:36:26.962006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.961972    5503 patch_prober.go:29] interesting pod/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:36:26.962389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:26.962038    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-gxjn5" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:36:27.654612 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:27.654578    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:27.792278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:27.792248    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" path="/var/lib/kubelet/pods/131bcd3e-372f-47ad-aca8-72a0a92d7d53/volumes"
Oct 30 22:36:27.792741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:27.792728    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" path="/var/lib/kubelet/pods/15ec8cd4-71b2-4643-a622-f557a2728ca6/volumes"
Oct 30 22:36:28.101588 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:36:28.101508    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:36:28.117810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:28.117783    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:28.927749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:28.927717    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:31.813010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:31.812979    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"]
Oct 30 22:36:31.813456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:31.813183    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy" containerID="cri-o://000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a" gracePeriod=30
Oct 30 22:36:31.815132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:31.815110    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"]
Oct 30 22:36:31.815293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:31.815273    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" containerID="cri-o://c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b" gracePeriod=30
Oct 30 22:36:33.542023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:33.541987    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:33.693226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:33.693193    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:36.906708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:36.906674    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 22:36:36.907112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:36.906729    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 22:36:37.152179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.152149    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:36:37.170769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.170745    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:36:37.199927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.199898    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-credential-socket\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.199938    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-token\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.199974    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-certs\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200003    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-envoy\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200065    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-podinfo\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200121    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-data\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200225    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-credential-socket" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.200350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200310    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-socket\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200352    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-certs" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.200522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200359    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-data" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.200522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200358    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-envoy\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200414    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47233975-6225-4d8e-a419-e203fc752e15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47233975-6225-4d8e-a419-e203fc752e15-istiod-ca-cert\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200450    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-istio-token\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-istio-token\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200485    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-certs\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200564    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47233975-6225-4d8e-a419-e203fc752e15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47233975-6225-4d8e-a419-e203fc752e15-istio-podinfo\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200583    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-socket" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200614    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-kube-api-access-94glv\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-kube-api-access-94glv\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200650    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-kube-api-access-vxz7c\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-kube-api-access-vxz7c\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200692    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-data\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200731    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-socket\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200773    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istiod-ca-cert\") pod \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\" (UID: \"77cf9753-70e3-4c77-84ca-688b9ef58cb9\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200819    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-credential-socket\") pod \"47233975-6225-4d8e-a419-e203fc752e15\" (UID: \"47233975-6225-4d8e-a419-e203fc752e15\") "
Oct 30 22:36:37.200858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200838    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-certs" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.201299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.200770    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47233975-6225-4d8e-a419-e203fc752e15-istiod-ca-cert" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:37.201299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201063    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-socket" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.201299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201285    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-credential-socket" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.201489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201380    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.201489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201403    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.201489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201419    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47233975-6225-4d8e-a419-e203fc752e15-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.201489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201434    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.201489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201448    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.201489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201461    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.201489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201476    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.201967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201621    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istiod-ca-cert" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:37.201967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.201677    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-data" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.202763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.202713    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-envoy" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.202891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.202816    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-envoy" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:37.203165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.203140    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-podinfo" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:37.203260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.203146    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-istio-token" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:37.203322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.203311    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-kube-api-access-vxz7c" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "kube-api-access-vxz7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:37.203792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.203766    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-token" pod "77cf9753-70e3-4c77-84ca-688b9ef58cb9" (UID: "77cf9753-70e3-4c77-84ca-688b9ef58cb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:37.203888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.203827    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47233975-6225-4d8e-a419-e203fc752e15-istio-podinfo" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:37.203934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.203896    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-kube-api-access-94glv" pod "47233975-6225-4d8e-a419-e203fc752e15" (UID: "47233975-6225-4d8e-a419-e203fc752e15"). InnerVolumeSpecName "kube-api-access-94glv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:37.302734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302701    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94glv\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-kube-api-access-94glv\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302729    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxz7c\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-kube-api-access-vxz7c\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302741    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302752    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302761    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302771    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302780    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302787    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77cf9753-70e3-4c77-84ca-688b9ef58cb9-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302795    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47233975-6225-4d8e-a419-e203fc752e15-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302803    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47233975-6225-4d8e-a419-e203fc752e15-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.302975 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.302811    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47233975-6225-4d8e-a419-e203fc752e15-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:37.512755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.512719    5503 generic.go:356] "Generic (PLEG): container finished" podID="47233975-6225-4d8e-a419-e203fc752e15" containerID="c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b" exitCode=0
Oct 30 22:36:37.512974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.512772    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"
Oct 30 22:36:37.512974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.512795    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" event={"ID":"47233975-6225-4d8e-a419-e203fc752e15","Type":"ContainerDied","Data":"c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b"}
Oct 30 22:36:37.512974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.512855    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z" event={"ID":"47233975-6225-4d8e-a419-e203fc752e15","Type":"ContainerDied","Data":"39ebb6fe45243c3ed844992d0ccb2c6a11ac049c1a6600e0675607ca455a58ff"}
Oct 30 22:36:37.512974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.512878    5503 scope.go:122] "RemoveContainer" containerID="c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b"
Oct 30 22:36:37.514523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.514499    5503 generic.go:356] "Generic (PLEG): container finished" podID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerID="000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a" exitCode=0
Oct 30 22:36:37.514659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.514560    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"
Oct 30 22:36:37.514659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.514580    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" event={"ID":"77cf9753-70e3-4c77-84ca-688b9ef58cb9","Type":"ContainerDied","Data":"000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a"}
Oct 30 22:36:37.514659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.514654    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf" event={"ID":"77cf9753-70e3-4c77-84ca-688b9ef58cb9","Type":"ContainerDied","Data":"050db5803f8a488f9fe3887d2d21f1352a3cdbf7e1804cbed40cbd3f4e417d5e"}
Oct 30 22:36:37.549966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.549943    5503 scope.go:122] "RemoveContainer" containerID="c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b"
Oct 30 22:36:37.550368 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:36:37.550344    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b\": container with ID starting with c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b not found: ID does not exist" containerID="c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b"
Oct 30 22:36:37.550458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.550376    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b"} err="failed to get container status \"c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b\": rpc error: code = NotFound desc = could not find container \"c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b\": container with ID starting with c629d28c34932ff1ff92f9a2de9c12600a8651f3c0af42c0750b4774bad7471b not found: ID does not exist"
Oct 30 22:36:37.550458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.550402    5503 scope.go:122] "RemoveContainer" containerID="000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a"
Oct 30 22:36:37.560038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.560016    5503 scope.go:122] "RemoveContainer" containerID="000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a"
Oct 30 22:36:37.560299 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:36:37.560281    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a\": container with ID starting with 000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a not found: ID does not exist" containerID="000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a"
Oct 30 22:36:37.560349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.560308    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a"} err="failed to get container status \"000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a\": rpc error: code = NotFound desc = could not find container \"000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a\": container with ID starting with 000ad7996181c52d4d66fdea2b746811a0083c6b6cb1a0a69a6316a9d2ee4e5a not found: ID does not exist"
Oct 30 22:36:37.705096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.705060    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"]
Oct 30 22:36:37.712286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.712249    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wdm7z"]
Oct 30 22:36:37.789790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.789702    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47233975-6225-4d8e-a419-e203fc752e15" path="/var/lib/kubelet/pods/47233975-6225-4d8e-a419-e203fc752e15/volumes"
Oct 30 22:36:37.836598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.836559    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"]
Oct 30 22:36:37.836898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:37.836880    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-28wqf"]
Oct 30 22:36:38.236806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:38.236733    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:39.789146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:39.789113    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" path="/var/lib/kubelet/pods/77cf9753-70e3-4c77-84ca-688b9ef58cb9/volumes"
Oct 30 22:36:40.753924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:40.753885    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:42.858460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:42.858366    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:45.600075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:45.600036    5503 generic.go:356] "Generic (PLEG): container finished" podID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerID="465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894" exitCode=0
Oct 30 22:36:45.600571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:45.600108    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" event={"ID":"e0f7f72d-4152-4235-a2e2-ab6404fbcf33","Type":"ContainerDied","Data":"465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894"}
Oct 30 22:36:45.600571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:45.600523    5503 scope.go:122] "RemoveContainer" containerID="465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894"
Oct 30 22:36:45.602063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:45.602026    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:36:45.602159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:45.602095    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:36:45.602199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:45.602157    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:36:46.607836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:46.607805    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" event={"ID":"e0f7f72d-4152-4235-a2e2-ab6404fbcf33","Type":"ContainerStarted","Data":"8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d"}
Oct 30 22:36:47.104943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:47.104907    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:36:47.105131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:47.105111    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:36:47.105179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:47.105160    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 22:36:47.340918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:47.340817    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:48.105237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:48.105204    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:36:48.105670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:48.105255    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 22:36:49.105203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:49.105167    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:36:49.105408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:49.105289    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 22:36:50.105107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:50.105065    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:36:50.105287 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:50.105157    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 22:36:51.104794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:51.104759    5503 patch_prober.go:29] interesting pod/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Oct 30 22:36:51.105198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:51.104816    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Oct 30 22:36:51.870985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:51.870948    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:52.110076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:52.110047    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:36:52.110472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:52.110316    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:36:52.111164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:52.111132    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:36:56.370130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:36:56.370098    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:36:58.100923 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:36:58.100885    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:37:00.950891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:00.950856    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:01.354752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:01.354631    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:01.516926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:01.516890    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:01.735402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:01.735371    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:02.638434 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638397    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"]
Oct 30 22:37:02.638908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638893    5503 cpu_manager.go:414] "containerMap: removing container" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy"
Oct 30 22:37:02.638908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638908    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy"
Oct 30 22:37:02.638995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638940    5503 cpu_manager.go:414] "containerMap: removing container" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy"
Oct 30 22:37:02.638995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638946    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy"
Oct 30 22:37:02.638995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638957    5503 cpu_manager.go:414] "containerMap: removing container" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy"
Oct 30 22:37:02.638995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638963    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy"
Oct 30 22:37:02.638995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638974    5503 cpu_manager.go:414] "containerMap: removing container" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy"
Oct 30 22:37:02.638995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.638978    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy"
Oct 30 22:37:02.639200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.639054    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="47233975-6225-4d8e-a419-e203fc752e15" containerName="istio-proxy"
Oct 30 22:37:02.639200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.639064    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="131bcd3e-372f-47ad-aca8-72a0a92d7d53" containerName="istio-proxy"
Oct 30 22:37:02.639200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.639070    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cf9753-70e3-4c77-84ca-688b9ef58cb9" containerName="istio-proxy"
Oct 30 22:37:02.639200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.639078    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ec8cd4-71b2-4643-a622-f557a2728ca6" containerName="istio-proxy"
Oct 30 22:37:02.642176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.642158    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.698125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.698091    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"]
Oct 30 22:37:02.745459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.745426    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:37:02.745459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.745446    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:37:02.745699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.745450    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:37:02.748029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.748008    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sn4d\" (UniqueName: \"kubernetes.io/projected/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-kube-api-access-8sn4d\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.748119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.748047    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.748256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.748234    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.749120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.749102    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:37:02.849400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.849364    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.849608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.849517    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sn4d\" (UniqueName: \"kubernetes.io/projected/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-kube-api-access-8sn4d\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.849608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.849552    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.850212 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.850189    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.851794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.851764    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.912126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.912092    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sn4d\" (UniqueName: \"kubernetes.io/projected/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-kube-api-access-8sn4d\") pod \"maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:02.956209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:02.956186    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:03.329991 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:37:03.329963    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b5c24a_24f6_4596_a1ef_993af75e9ac7.slice/crio-cc669fbff236cf36b30eb32abe008984561f3874e7b87d77bdb40fac2572738b WatchSource:0}: Error finding container cc669fbff236cf36b30eb32abe008984561f3874e7b87d77bdb40fac2572738b: Status 404 returned error can't find the container with id cc669fbff236cf36b30eb32abe008984561f3874e7b87d77bdb40fac2572738b
Oct 30 22:37:03.332345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:03.332323    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"]
Oct 30 22:37:03.737629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:03.737586    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm" event={"ID":"b8b5c24a-24f6-4596-a1ef-993af75e9ac7","Type":"ContainerStarted","Data":"4765d94af54a1722286b3bf095b565027017a9add8ff7d1ea528bca833f3fd59"}
Oct 30 22:37:03.738110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:03.737680    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm" event={"ID":"b8b5c24a-24f6-4596-a1ef-993af75e9ac7","Type":"ContainerStarted","Data":"cc669fbff236cf36b30eb32abe008984561f3874e7b87d77bdb40fac2572738b"}
Oct 30 22:37:03.855849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:03.855797    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm" podStartSLOduration=1.855783242 podStartE2EDuration="1.855783242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:03.855406833 +0000 UTC m=+3000.189983602" watchObservedRunningTime="2026-10-30 22:37:03.855783242 +0000 UTC m=+3000.190360010"
Oct 30 22:37:04.157211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:04.157182    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:04.285217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:04.285188    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:37:04.296925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:04.296904    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:37:04.745397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:04.745365    5503 generic.go:356] "Generic (PLEG): container finished" podID="b8b5c24a-24f6-4596-a1ef-993af75e9ac7" containerID="4765d94af54a1722286b3bf095b565027017a9add8ff7d1ea528bca833f3fd59" exitCode=0
Oct 30 22:37:04.745822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:04.745450    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm" event={"ID":"b8b5c24a-24f6-4596-a1ef-993af75e9ac7","Type":"ContainerDied","Data":"4765d94af54a1722286b3bf095b565027017a9add8ff7d1ea528bca833f3fd59"}
Oct 30 22:37:04.913295 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:04.913261    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:37:05.570353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.570316    5503 scope.go:122] "RemoveContainer" containerID="28b542a7d750fedadce33c118a568777f93e38d88bd140ceb31c35dc08f0af77"
Oct 30 22:37:05.579550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.579525    5503 scope.go:122] "RemoveContainer" containerID="775d1659a17c7371617883fcc3143909b1a6e4c343a1e9ecbf92dffeb3169c9d"
Oct 30 22:37:05.588062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.588045    5503 scope.go:122] "RemoveContainer" containerID="79e23aea44949a0b3438ed5e730d1aae186c31a2a42975a504410b47a0afa646"
Oct 30 22:37:05.794163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.794130    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:05.929471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.929441    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:05.992216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.992187    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-openshift-ca\") pod \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") "
Oct 30 22:37:05.992382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.992229    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-xks-ca\") pod \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") "
Oct 30 22:37:05.992382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.992255    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-kube-api-access-8sn4d\" (UniqueName: \"kubernetes.io/projected/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-kube-api-access-8sn4d\") pod \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\" (UID: \"b8b5c24a-24f6-4596-a1ef-993af75e9ac7\") "
Oct 30 22:37:05.992559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.992536    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-openshift-ca" pod "b8b5c24a-24f6-4596-a1ef-993af75e9ac7" (UID: "b8b5c24a-24f6-4596-a1ef-993af75e9ac7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:05.992823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.992799    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:05.994164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.994141    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-xks-ca" pod "b8b5c24a-24f6-4596-a1ef-993af75e9ac7" (UID: "b8b5c24a-24f6-4596-a1ef-993af75e9ac7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:37:05.994346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:05.994332    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-kube-api-access-8sn4d" pod "b8b5c24a-24f6-4596-a1ef-993af75e9ac7" (UID: "b8b5c24a-24f6-4596-a1ef-993af75e9ac7"). InnerVolumeSpecName "kube-api-access-8sn4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:06.093777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:06.093747    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:06.093777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:06.093775    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sn4d\" (UniqueName: \"kubernetes.io/projected/b8b5c24a-24f6-4596-a1ef-993af75e9ac7-kube-api-access-8sn4d\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:06.757995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:06.757966    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"
Oct 30 22:37:06.758175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:06.758005    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm" event={"ID":"b8b5c24a-24f6-4596-a1ef-993af75e9ac7","Type":"ContainerDied","Data":"cc669fbff236cf36b30eb32abe008984561f3874e7b87d77bdb40fac2572738b"}
Oct 30 22:37:06.758175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:06.758058    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc669fbff236cf36b30eb32abe008984561f3874e7b87d77bdb40fac2572738b"
Oct 30 22:37:10.512255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:10.512219    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:14.565456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:14.565416    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:14.721045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:14.720998    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:15.068957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:15.068924    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:19.454429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:19.454391    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:19.710567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:19.710486    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:24.372735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:24.372692    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:26.800814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:26.800780    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:28.019546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:28.019511    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:28.099012 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:37:28.098975    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:37:28.928133 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:28.928092    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:33.472095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:33.472062    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:34.223592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:34.223557    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:37.953413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:37.953360    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:40.735125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:40.735088    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"]
Oct 30 22:37:40.735560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:40.735320    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy" containerID="cri-o://7905855a62daa084e08caad08106819f680853f127a10895db081e03ec1dc99e" gracePeriod=30
Oct 30 22:37:42.521125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:42.521084    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:43.187677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:43.187626    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:37:45.983189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:45.983160    5503 generic.go:356] "Generic (PLEG): container finished" podID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerID="7905855a62daa084e08caad08106819f680853f127a10895db081e03ec1dc99e" exitCode=0
Oct 30 22:37:45.983556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:45.983236    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" event={"ID":"c4f9080d-92e8-459f-a858-261e5bb09d3d","Type":"ContainerDied","Data":"7905855a62daa084e08caad08106819f680853f127a10895db081e03ec1dc99e"}
Oct 30 22:37:46.005167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.005113    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:37:46.073009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.072975    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-token\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-token\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073028    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-credential-socket\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073078    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-kube-api-access-6n8xg\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-kube-api-access-6n8xg\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073142    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-envoy\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073171    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f9080d-92e8-459f-a858-261e5bb09d3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f9080d-92e8-459f-a858-261e5bb09d3d-istiod-ca-cert\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073212    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-podinfo\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073248    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-socket\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073291    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-data\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073352    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-certs\") pod \"c4f9080d-92e8-459f-a858-261e5bb09d3d\" (UID: \"c4f9080d-92e8-459f-a858-261e5bb09d3d\") "
Oct 30 22:37:46.073443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073377    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-credential-socket" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:46.073875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073606    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-socket" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:46.073875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073665    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-data" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:46.073875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073836    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-certs" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:46.074049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.073928    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f9080d-92e8-459f-a858-261e5bb09d3d-istiod-ca-cert" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:37:46.074310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.074287    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4f9080d-92e8-459f-a858-261e5bb09d3d-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.074387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.074316    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.074387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.074332    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.074387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.074345    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.074387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.074360    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.075407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.075376    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-token" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:46.075521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.075439    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-podinfo" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:37:46.075521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.075493    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-envoy" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:37:46.075521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.075496    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-kube-api-access-6n8xg" pod "c4f9080d-92e8-459f-a858-261e5bb09d3d" (UID: "c4f9080d-92e8-459f-a858-261e5bb09d3d"). InnerVolumeSpecName "kube-api-access-6n8xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:37:46.175540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.175508    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.175540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.175536    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8xg\" (UniqueName: \"kubernetes.io/projected/c4f9080d-92e8-459f-a858-261e5bb09d3d-kube-api-access-6n8xg\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.175540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.175548    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.175791 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.175557    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4f9080d-92e8-459f-a858-261e5bb09d3d-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:37:46.990835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.990803    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"
Oct 30 22:37:46.991340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.990847    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg" event={"ID":"c4f9080d-92e8-459f-a858-261e5bb09d3d","Type":"ContainerDied","Data":"0fb23225dcc95edece0eddbb4ba8565f923a414a7ad1f6b6126bab8bd0a7926d"}
Oct 30 22:37:46.991340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:46.990907    5503 scope.go:122] "RemoveContainer" containerID="7905855a62daa084e08caad08106819f680853f127a10895db081e03ec1dc99e"
Oct 30 22:37:47.147558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:47.147527    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:47.206700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:47.206668    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"]
Oct 30 22:37:47.257471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:47.257391    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6f54779779-58nlg"]
Oct 30 22:37:47.788544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:47.788510    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" path="/var/lib/kubelet/pods/c4f9080d-92e8-459f-a858-261e5bb09d3d/volumes"
Oct 30 22:37:51.576440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:51.576407    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:56.154267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:37:56.154225    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:37:58.099364 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:37:58.099327    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:38:00.818383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:00.818352    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:05.336775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:05.336699    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:05.639127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:05.639043    5503 scope.go:122] "RemoveContainer" containerID="d895cfeebd94c4e766ebc9ace8c4bd2406d6193839309303f25fdc5eaf07e818"
Oct 30 22:38:09.871752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:09.871718    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:13.853199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:13.853120    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:15.252825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:15.252794    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:15.575147 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:15.575069    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:15.731043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:15.731011    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:15.945901 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:15.945871    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"]
Oct 30 22:38:15.995846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:15.995812    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-28f124-lwlmm"]
Oct 30 22:38:16.466545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:16.466502    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:17.789286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:17.789249    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b5c24a-24f6-4596-a1ef-993af75e9ac7" path="/var/lib/kubelet/pods/b8b5c24a-24f6-4596-a1ef-993af75e9ac7/volumes"
Oct 30 22:38:18.730841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:18.730796    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:20.319401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:20.319371    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:38:20.347803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:20.347772    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:23.542321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:23.542279    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:23.728979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:23.728947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:23.883623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:23.883541    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"]
Oct 30 22:38:23.883780 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:23.883756    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy" containerID="cri-o://dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c" gracePeriod=30
Oct 30 22:38:23.894541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:23.894511    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:25.019594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:25.019562    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:27.124279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:27.124251    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:28.090852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:28.090816    5503 patch_prober.go:29] interesting pod/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:38:28.091025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:28.090883    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:38:28.099470 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:38:28.099437    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:38:28.359991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:28.359908    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:29.147659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.147578    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:38:29.244383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.244349    5503 generic.go:356] "Generic (PLEG): container finished" podID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerID="dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c" exitCode=0
Oct 30 22:38:29.244576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.244406    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"
Oct 30 22:38:29.244576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.244429    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" event={"ID":"3fdddc2d-a58b-44b0-afed-cc12f2018c75","Type":"ContainerDied","Data":"dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c"}
Oct 30 22:38:29.244576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.244481    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc" event={"ID":"3fdddc2d-a58b-44b0-afed-cc12f2018c75","Type":"ContainerDied","Data":"bd6fb75bae0565260fb10c3adaeca9e85bccde0af53327570f82a113161ba223"}
Oct 30 22:38:29.244576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.244504    5503 scope.go:122] "RemoveContainer" containerID="dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c"
Oct 30 22:38:29.254054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.254039    5503 scope.go:122] "RemoveContainer" containerID="dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c"
Oct 30 22:38:29.254308 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:38:29.254289    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c\": container with ID starting with dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c not found: ID does not exist" containerID="dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c"
Oct 30 22:38:29.254363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.254315    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c"} err="failed to get container status \"dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c\": rpc error: code = NotFound desc = could not find container \"dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c\": container with ID starting with dc82418afc0c9dcb72ee89c3dd20bf8f535a0a0ca440690bc27934ee11302e2c not found: ID does not exist"
Oct 30 22:38:29.284874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.284849    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-podinfo\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.284907    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istiod-ca-cert\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.284936    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-kube-api-access-nbfrt\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-kube-api-access-nbfrt\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.284981    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-envoy\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285079    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-certs\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285119    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-credential-socket\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285220    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-socket\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285273    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-data\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285350    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-token\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-token\") pod \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\" (UID: \"3fdddc2d-a58b-44b0-afed-cc12f2018c75\") "
Oct 30 22:38:29.285428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285364    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-certs" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:29.285531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285437    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-credential-socket" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:29.285531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285454    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istiod-ca-cert" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:29.285531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285483    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-socket" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:29.285723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285577    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-data" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:29.285848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285826    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.285997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.285981    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.286063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.286001    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.286063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.286015    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.286063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.286029    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.287252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.287228    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-podinfo" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:38:29.287357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.287260    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-kube-api-access-nbfrt" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "kube-api-access-nbfrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:29.287687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.287665    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-envoy" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:29.287687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.287677    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-token" pod "3fdddc2d-a58b-44b0-afed-cc12f2018c75" (UID: "3fdddc2d-a58b-44b0-afed-cc12f2018c75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:29.322419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.322389    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"]
Oct 30 22:38:29.322876 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.322861    5503 cpu_manager.go:414] "containerMap: removing container" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy"
Oct 30 22:38:29.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.322877    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy"
Oct 30 22:38:29.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.322896    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy"
Oct 30 22:38:29.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.322901    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy"
Oct 30 22:38:29.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.322920    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b8b5c24a-24f6-4596-a1ef-993af75e9ac7" containerName="revoke-keys"
Oct 30 22:38:29.322960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.322925    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b5c24a-24f6-4596-a1ef-993af75e9ac7" containerName="revoke-keys"
Oct 30 22:38:29.323220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.323007    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" containerName="istio-proxy"
Oct 30 22:38:29.323220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.323017    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b5c24a-24f6-4596-a1ef-993af75e9ac7" containerName="revoke-keys"
Oct 30 22:38:29.323220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.323030    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f9080d-92e8-459f-a858-261e5bb09d3d" containerName="istio-proxy"
Oct 30 22:38:29.327435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.327418    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.381235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.381196    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:38:29.381731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.381281    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:38:29.382033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.382007    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:38:29.382170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.382099    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:38:29.382799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.382779    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"]
Oct 30 22:38:29.387145 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.387118    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.387252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.387149    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.387252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.387165    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfrt\" (UniqueName: \"kubernetes.io/projected/3fdddc2d-a58b-44b0-afed-cc12f2018c75-kube-api-access-nbfrt\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.387252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.387180    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdddc2d-a58b-44b0-afed-cc12f2018c75-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:29.488442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.488404    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.488687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.488474    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.488687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.488545    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj747\" (UniqueName: \"kubernetes.io/projected/451b0d73-a4ae-41c9-b52a-a042d73233d9-kube-api-access-tj747\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.589662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.589605    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.589845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.589706    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.589845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.589742    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj747\" (UniqueName: \"kubernetes.io/projected/451b0d73-a4ae-41c9-b52a-a042d73233d9-kube-api-access-tj747\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.590271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.590249    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.592021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.591999    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.655023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.654995    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj747\" (UniqueName: \"kubernetes.io/projected/451b0d73-a4ae-41c9-b52a-a042d73233d9-kube-api-access-tj747\") pod \"maas-api-revoke-sub-7e03ebd7-x8q62\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:29.681577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.681537    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"]
Oct 30 22:38:29.686361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.686327    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-28f124-openshift-default-98987bd97-f6fjc"]
Oct 30 22:38:29.714540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.714510    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:29.788453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.788378    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdddc2d-a58b-44b0-afed-cc12f2018c75" path="/var/lib/kubelet/pods/3fdddc2d-a58b-44b0-afed-cc12f2018c75/volumes"
Oct 30 22:38:29.940942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:29.940916    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:30.311649 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:38:30.311607    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451b0d73_a4ae_41c9_b52a_a042d73233d9.slice/crio-1dc1cf07dd2daf87ed232a488bc68e4b7bf4c784aaa2ced31561998262ae55e6 WatchSource:0}: Error finding container 1dc1cf07dd2daf87ed232a488bc68e4b7bf4c784aaa2ced31561998262ae55e6: Status 404 returned error can't find the container with id 1dc1cf07dd2daf87ed232a488bc68e4b7bf4c784aaa2ced31561998262ae55e6
Oct 30 22:38:30.314141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:30.314117    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"]
Oct 30 22:38:31.257944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:31.257907    5503 generic.go:356] "Generic (PLEG): container finished" podID="451b0d73-a4ae-41c9-b52a-a042d73233d9" containerID="456565cb40635acf0bee437b89d6ca0634fadd9ea5746ccf113ddea0d1ff29eb" exitCode=0
Oct 30 22:38:31.258315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:31.257984    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62" event={"ID":"451b0d73-a4ae-41c9-b52a-a042d73233d9","Type":"ContainerDied","Data":"456565cb40635acf0bee437b89d6ca0634fadd9ea5746ccf113ddea0d1ff29eb"}
Oct 30 22:38:31.258315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:31.258033    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62" event={"ID":"451b0d73-a4ae-41c9-b52a-a042d73233d9","Type":"ContainerStarted","Data":"1dc1cf07dd2daf87ed232a488bc68e4b7bf4c784aaa2ced31561998262ae55e6"}
Oct 30 22:38:32.405655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.405609    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:32.524753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.524669    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451b0d73-a4ae-41c9-b52a-a042d73233d9-kube-api-access-tj747\" (UniqueName: \"kubernetes.io/projected/451b0d73-a4ae-41c9-b52a-a042d73233d9-kube-api-access-tj747\") pod \"451b0d73-a4ae-41c9-b52a-a042d73233d9\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") "
Oct 30 22:38:32.524910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.524766    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-xks-ca\") pod \"451b0d73-a4ae-41c9-b52a-a042d73233d9\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") "
Oct 30 22:38:32.525013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.524964    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-openshift-ca\") pod \"451b0d73-a4ae-41c9-b52a-a042d73233d9\" (UID: \"451b0d73-a4ae-41c9-b52a-a042d73233d9\") "
Oct 30 22:38:32.525433 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.525396    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-openshift-ca" pod "451b0d73-a4ae-41c9-b52a-a042d73233d9" (UID: "451b0d73-a4ae-41c9-b52a-a042d73233d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:32.527109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.527079    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451b0d73-a4ae-41c9-b52a-a042d73233d9-kube-api-access-tj747" pod "451b0d73-a4ae-41c9-b52a-a042d73233d9" (UID: "451b0d73-a4ae-41c9-b52a-a042d73233d9"). InnerVolumeSpecName "kube-api-access-tj747". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:32.527217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.527182    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-xks-ca" pod "451b0d73-a4ae-41c9-b52a-a042d73233d9" (UID: "451b0d73-a4ae-41c9-b52a-a042d73233d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:32.626613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.626576    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:32.626613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.626608    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/451b0d73-a4ae-41c9-b52a-a042d73233d9-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:32.626613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:32.626618    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj747\" (UniqueName: \"kubernetes.io/projected/451b0d73-a4ae-41c9-b52a-a042d73233d9-kube-api-access-tj747\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:38:33.270291 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:33.270260    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"
Oct 30 22:38:33.270471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:33.270260    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62" event={"ID":"451b0d73-a4ae-41c9-b52a-a042d73233d9","Type":"ContainerDied","Data":"1dc1cf07dd2daf87ed232a488bc68e4b7bf4c784aaa2ced31561998262ae55e6"}
Oct 30 22:38:33.270471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:33.270385    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dc1cf07dd2daf87ed232a488bc68e4b7bf4c784aaa2ced31561998262ae55e6"
Oct 30 22:38:34.230557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:34.230527    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:38.817281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:38.817253    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:41.637156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:41.637117    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:43.221803 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:43.221767    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:43.386944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:43.386904    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:43.664363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:43.664331    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:47.417056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:47.417023    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:47.652649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:47.652601    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:47.853754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:47.853668    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:48.515882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:48.515847    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:50.683456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:50.683412    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:50.867744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:50.867704    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:51.096485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.096404    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"]
Oct 30 22:38:51.096933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.096917    5503 cpu_manager.go:414] "containerMap: removing container" podUID="451b0d73-a4ae-41c9-b52a-a042d73233d9" containerName="revoke-keys"
Oct 30 22:38:51.097001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.096933    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451b0d73-a4ae-41c9-b52a-a042d73233d9" containerName="revoke-keys"
Oct 30 22:38:51.097040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.097034    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="451b0d73-a4ae-41c9-b52a-a042d73233d9" containerName="revoke-keys"
Oct 30 22:38:51.100359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.100336    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.154924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.154883    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"]
Oct 30 22:38:51.213025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.212986    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-cbe3a9\""
Oct 30 22:38:51.213650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.213610    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-cbe3a9-dockercfg-knlfq\""
Oct 30 22:38:51.257520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.257475    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48r59\" (UniqueName: \"kubernetes.io/projected/184102c8-ecdc-4c5e-b07d-868cbcd4324d-kube-api-access-48r59\") pod \"payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.257882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.257856    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184102c8-ecdc-4c5e-b07d-868cbcd4324d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.270757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.270723    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"]
Oct 30 22:38:51.275808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.275784    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.303628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.303592    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"]
Oct 30 22:38:51.358906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.358825    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.358906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.358865    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlh7d\" (UniqueName: \"kubernetes.io/projected/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-kube-api-access-qlh7d\") pod \"payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.359107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.358958    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48r59\" (UniqueName: \"kubernetes.io/projected/184102c8-ecdc-4c5e-b07d-868cbcd4324d-kube-api-access-48r59\") pod \"payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.359107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.359053    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184102c8-ecdc-4c5e-b07d-868cbcd4324d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.359733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.359716    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184102c8-ecdc-4c5e-b07d-868cbcd4324d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.422553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.422524    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48r59\" (UniqueName: \"kubernetes.io/projected/184102c8-ecdc-4c5e-b07d-868cbcd4324d-kube-api-access-48r59\") pod \"payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.459792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.459758    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.459950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.459797    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlh7d\" (UniqueName: \"kubernetes.io/projected/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-kube-api-access-qlh7d\") pod \"payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.460459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.460437    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.523190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.523160    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlh7d\" (UniqueName: \"kubernetes.io/projected/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-kube-api-access-qlh7d\") pod \"payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.592950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.592911    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:51.714010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.713977    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:51.741257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.741227    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"]
Oct 30 22:38:51.745312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.745287    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.801810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.801780    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e65019-openshift-default-dockercfg-jjq2k\""
Oct 30 22:38:51.805677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.805611    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"]
Oct 30 22:38:51.864432 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864397    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrkd\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-kube-api-access-5zrkd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864493    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864579    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864623    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864667    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864765    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864810    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864880    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.864951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.864936    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.961600 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:38:51.961573    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d6bc7_9fbd_4c48_add2_5fa35e9bdf4b.slice/crio-d9d554f9f4e190b38d10b3fe85d2f3c5214b9bdfddbbe5bdfc1acb977ae8b228 WatchSource:0}: Error finding container d9d554f9f4e190b38d10b3fe85d2f3c5214b9bdfddbbe5bdfc1acb977ae8b228: Status 404 returned error can't find the container with id d9d554f9f4e190b38d10b3fe85d2f3c5214b9bdfddbbe5bdfc1acb977ae8b228
Oct 30 22:38:51.964086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.964065    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"]
Oct 30 22:38:51.966348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966329    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966368    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966408    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966441    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrkd\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-kube-api-access-5zrkd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966463    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966506    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966550    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966591    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966629    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966776    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-certs\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.966896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.966850    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.967128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.967041    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-data\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.967128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.967084    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istiod-ca-cert\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.967241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.967222    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-credential-socket\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.969105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.969081    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-envoy\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:51.969341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:51.969312    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-podinfo\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:52.029041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.029009    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-token\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:52.029162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.029142    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrkd\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-kube-api-access-5zrkd\") pod \"e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") " pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:52.064395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.064362    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:52.088909 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:38:52.088878    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184102c8_ecdc_4c5e_b07d_868cbcd4324d.slice/crio-1c7f615f68ac22deb1f5bbd8bf8a655d6f3e3f182914ad71422a02ba2e954ed6 WatchSource:0}: Error finding container 1c7f615f68ac22deb1f5bbd8bf8a655d6f3e3f182914ad71422a02ba2e954ed6: Status 404 returned error can't find the container with id 1c7f615f68ac22deb1f5bbd8bf8a655d6f3e3f182914ad71422a02ba2e954ed6
Oct 30 22:38:52.090830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.090800    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"]
Oct 30 22:38:52.396624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.396539    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw" event={"ID":"184102c8-ecdc-4c5e-b07d-868cbcd4324d","Type":"ContainerStarted","Data":"a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0"}
Oct 30 22:38:52.396624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.396591    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw" event={"ID":"184102c8-ecdc-4c5e-b07d-868cbcd4324d","Type":"ContainerStarted","Data":"1c7f615f68ac22deb1f5bbd8bf8a655d6f3e3f182914ad71422a02ba2e954ed6"}
Oct 30 22:38:52.396856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.396679    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:38:52.398482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.398454    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh" event={"ID":"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b","Type":"ContainerStarted","Data":"57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58"}
Oct 30 22:38:52.398629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.398496    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh" event={"ID":"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b","Type":"ContainerStarted","Data":"d9d554f9f4e190b38d10b3fe85d2f3c5214b9bdfddbbe5bdfc1acb977ae8b228"}
Oct 30 22:38:52.398629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.398518    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:38:52.449076 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:38:52.449041    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19dd129c_14f1_4dd1_984a_f08dc42f64fd.slice/crio-a8b856c2277d3bce7bacfc4745f5c7b5e8ba0098d123abf7f614abe86d842fa8 WatchSource:0}: Error finding container a8b856c2277d3bce7bacfc4745f5c7b5e8ba0098d123abf7f614abe86d842fa8: Status 404 returned error can't find the container with id a8b856c2277d3bce7bacfc4745f5c7b5e8ba0098d123abf7f614abe86d842fa8
Oct 30 22:38:52.451462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.451436    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"]
Oct 30 22:38:52.452135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.452090    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:38:52.452234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.452152    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:38:52.452234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.452185    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:38:52.516406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.516360    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw" podStartSLOduration=1.51634449 podStartE2EDuration="1.51634449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:52.514362189 +0000 UTC m=+3108.848938956" watchObservedRunningTime="2026-10-30 22:38:52.51634449 +0000 UTC m=+3108.850921260"
Oct 30 22:38:52.638814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:52.638741    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh" podStartSLOduration=1.63872069 podStartE2EDuration="1.63872069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:52.636991542 +0000 UTC m=+3108.971568312" watchObservedRunningTime="2026-10-30 22:38:52.63872069 +0000 UTC m=+3108.973297460"
Oct 30 22:38:53.273397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:53.273363    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:53.408161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:53.408119    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" event={"ID":"19dd129c-14f1-4dd1-984a-f08dc42f64fd","Type":"ContainerStarted","Data":"9d28507300ace1bb24f93e8b0ac6a125ec258b4bcc4bc26e0b8942e1947f28f7"}
Oct 30 22:38:53.408377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:53.408186    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" event={"ID":"19dd129c-14f1-4dd1-984a-f08dc42f64fd","Type":"ContainerStarted","Data":"a8b856c2277d3bce7bacfc4745f5c7b5e8ba0098d123abf7f614abe86d842fa8"}
Oct 30 22:38:53.432644 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:53.432584    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:53.537720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:53.534678    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podStartSLOduration=2.534658028 podStartE2EDuration="2.534658028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:53.533102908 +0000 UTC m=+3109.867679699" watchObservedRunningTime="2026-10-30 22:38:53.534658028 +0000 UTC m=+3109.869234800"
Oct 30 22:38:54.064993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:54.064956    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:38:54.066609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:54.066583    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:38:54.066766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:54.066656    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 22:38:55.064802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:55.064772    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:38:55.065218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:55.064818    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 22:38:56.065425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:56.065392    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:38:56.065844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:56.065454    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 22:38:57.065555 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:57.065520    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:38:57.065971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:57.065578    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 22:38:57.914615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:57.914585    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:38:58.065542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:58.065508    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:38:58.065742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:58.065563    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 22:38:58.098124 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:38:58.098088    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:38:59.065159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:59.065126    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:38:59.065380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:38:59.065178    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 22:39:00.074299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:00.074265    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:39:00.074779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:00.074676    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:39:00.075515 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:00.075490    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:39:02.424516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:02.424482    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:03.411089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:03.411057    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:39:03.411247 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:03.411132    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:39:06.961615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:06.961581    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:11.508493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:11.508458    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:15.909872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:15.909832    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:39:16.044326 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:16.044289    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:20.715583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:20.715548    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:25.204104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:25.204054    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:27.462625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:27.462588    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:28.099229 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:39:28.099184    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:39:28.698438 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:28.698369    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:29.704380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:29.704345    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:34.053993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:34.053956    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"]
Oct 30 22:39:34.104286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:34.104260    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b6e88e7-spkvq"]
Oct 30 22:39:35.192546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:35.192511    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:35.788453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:35.788421    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630cbefe-5690-49e4-8450-424c84c50d2c" path="/var/lib/kubelet/pods/630cbefe-5690-49e4-8450-424c84c50d2c/volumes"
Oct 30 22:39:38.376064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:38.376025    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:38.538373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:38.538333    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:38.843436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:38.843403    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:43.417011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:43.416980    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:47.233319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.233278    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"]
Oct 30 22:39:47.237475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.237456    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.327597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.327558    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxh9q\" (UniqueName: \"kubernetes.io/projected/997d6fa9-8222-42eb-b4a1-ac9bf0290185-kube-api-access-xxh9q\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.327783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.327601    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.327841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.327826    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.341855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.341819    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:39:47.341855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.341849    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:47.342069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.341873    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:47.342069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.341922    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:47.344110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.344091    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"]
Oct 30 22:39:47.428742 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.428707    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.428912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.428818    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.429005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.428910    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxh9q\" (UniqueName: \"kubernetes.io/projected/997d6fa9-8222-42eb-b4a1-ac9bf0290185-kube-api-access-xxh9q\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.429477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.429452    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.431135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.431109    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.491972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.491895    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxh9q\" (UniqueName: \"kubernetes.io/projected/997d6fa9-8222-42eb-b4a1-ac9bf0290185-kube-api-access-xxh9q\") pod \"maas-api-revoke-sub-cb6c7e75-hjfcz\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.550616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.550580    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:47.920048 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:39:47.920002    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997d6fa9_8222_42eb_b4a1_ac9bf0290185.slice/crio-a861b9a0929e1c31ba5e82ebf19145d4b1ed27a0d527475e6e1a1dc3d7c47717 WatchSource:0}: Error finding container a861b9a0929e1c31ba5e82ebf19145d4b1ed27a0d527475e6e1a1dc3d7c47717: Status 404 returned error can't find the container with id a861b9a0929e1c31ba5e82ebf19145d4b1ed27a0d527475e6e1a1dc3d7c47717
Oct 30 22:39:47.922083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:47.922058    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"]
Oct 30 22:39:48.010243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:48.010214    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:48.764014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:48.763980    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz" event={"ID":"997d6fa9-8222-42eb-b4a1-ac9bf0290185","Type":"ContainerStarted","Data":"f9b392c1dffc23d77f446ab7b821e50a4b27fe0cb2d7da321ce06233d7fe380e"}
Oct 30 22:39:48.764410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:48.764036    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz" event={"ID":"997d6fa9-8222-42eb-b4a1-ac9bf0290185","Type":"ContainerStarted","Data":"a861b9a0929e1c31ba5e82ebf19145d4b1ed27a0d527475e6e1a1dc3d7c47717"}
Oct 30 22:39:48.882650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:48.882585    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz" podStartSLOduration=1.882572433 podStartE2EDuration="1.882572433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:48.881012914 +0000 UTC m=+3165.215589683" watchObservedRunningTime="2026-10-30 22:39:48.882572433 +0000 UTC m=+3165.217149235"
Oct 30 22:39:49.770998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:49.770954    5503 generic.go:356] "Generic (PLEG): container finished" podID="997d6fa9-8222-42eb-b4a1-ac9bf0290185" containerID="f9b392c1dffc23d77f446ab7b821e50a4b27fe0cb2d7da321ce06233d7fe380e" exitCode=0
Oct 30 22:39:49.771457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:49.771037    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz" event={"ID":"997d6fa9-8222-42eb-b4a1-ac9bf0290185","Type":"ContainerDied","Data":"f9b392c1dffc23d77f446ab7b821e50a4b27fe0cb2d7da321ce06233d7fe380e"}
Oct 30 22:39:50.913820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:50.913794    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:51.069458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.069383    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-xks-ca\") pod \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") "
Oct 30 22:39:51.069611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.069530    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997d6fa9-8222-42eb-b4a1-ac9bf0290185-kube-api-access-xxh9q\" (UniqueName: \"kubernetes.io/projected/997d6fa9-8222-42eb-b4a1-ac9bf0290185-kube-api-access-xxh9q\") pod \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") "
Oct 30 22:39:51.069611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.069571    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-openshift-ca\") pod \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\" (UID: \"997d6fa9-8222-42eb-b4a1-ac9bf0290185\") "
Oct 30 22:39:51.069922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.069892    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-openshift-ca" pod "997d6fa9-8222-42eb-b4a1-ac9bf0290185" (UID: "997d6fa9-8222-42eb-b4a1-ac9bf0290185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:51.071436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.071413    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-xks-ca" pod "997d6fa9-8222-42eb-b4a1-ac9bf0290185" (UID: "997d6fa9-8222-42eb-b4a1-ac9bf0290185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:51.071526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.071495    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997d6fa9-8222-42eb-b4a1-ac9bf0290185-kube-api-access-xxh9q" pod "997d6fa9-8222-42eb-b4a1-ac9bf0290185" (UID: "997d6fa9-8222-42eb-b4a1-ac9bf0290185"). InnerVolumeSpecName "kube-api-access-xxh9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:51.170898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.170863    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:51.170898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.170898    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxh9q\" (UniqueName: \"kubernetes.io/projected/997d6fa9-8222-42eb-b4a1-ac9bf0290185-kube-api-access-xxh9q\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:51.171106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.170910    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d6fa9-8222-42eb-b4a1-ac9bf0290185-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:51.784399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.784368    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"
Oct 30 22:39:51.788651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.788610    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz" event={"ID":"997d6fa9-8222-42eb-b4a1-ac9bf0290185","Type":"ContainerDied","Data":"a861b9a0929e1c31ba5e82ebf19145d4b1ed27a0d527475e6e1a1dc3d7c47717"}
Oct 30 22:39:51.788784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:51.788684    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a861b9a0929e1c31ba5e82ebf19145d4b1ed27a0d527475e6e1a1dc3d7c47717"
Oct 30 22:39:52.487972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:52.487941    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:52.656891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:52.656855    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:57.234383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.234347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:39:57.948491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.948459    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"]
Oct 30 22:39:57.949001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.948984    5503 cpu_manager.go:414] "containerMap: removing container" podUID="997d6fa9-8222-42eb-b4a1-ac9bf0290185" containerName="revoke-keys"
Oct 30 22:39:57.949063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.949002    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997d6fa9-8222-42eb-b4a1-ac9bf0290185" containerName="revoke-keys"
Oct 30 22:39:57.949107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.949079    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="997d6fa9-8222-42eb-b4a1-ac9bf0290185" containerName="revoke-keys"
Oct 30 22:39:57.952404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.952388    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:57.978670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.978622    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72hr4\" (UniqueName: \"kubernetes.io/projected/6b6be266-a9f8-4938-8b03-1c258ebd83ce-kube-api-access-72hr4\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:57.978819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.978684    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:57.978886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:57.978867    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.006830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.006802    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"]
Oct 30 22:39:58.055187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.055158    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:39:58.055369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.055251    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:39:58.055369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.055270    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:39:58.056695 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.056680    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:39:58.079957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.079927    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.080093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.080055    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.080161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.080149    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72hr4\" (UniqueName: \"kubernetes.io/projected/6b6be266-a9f8-4938-8b03-1c258ebd83ce-kube-api-access-72hr4\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.080843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.080817    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.082234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.082212    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.099211 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:39:58.099184    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:39:58.142777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.142750    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72hr4\" (UniqueName: \"kubernetes.io/projected/6b6be266-a9f8-4938-8b03-1c258ebd83ce-kube-api-access-72hr4\") pod \"maas-api-revoke-sub-efca9136-ngtz6\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.265268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.265187    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:39:58.639130 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:39:58.639086    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6be266_a9f8_4938_8b03_1c258ebd83ce.slice/crio-4a70122b41a46f6b01e62b2c5ddace0d1ea92ca435ced0fda3f153d31fc70e25 WatchSource:0}: Error finding container 4a70122b41a46f6b01e62b2c5ddace0d1ea92ca435ced0fda3f153d31fc70e25: Status 404 returned error can't find the container with id 4a70122b41a46f6b01e62b2c5ddace0d1ea92ca435ced0fda3f153d31fc70e25
Oct 30 22:39:58.641584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.641557    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"]
Oct 30 22:39:58.824726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:58.824694    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6" event={"ID":"6b6be266-a9f8-4938-8b03-1c258ebd83ce","Type":"ContainerStarted","Data":"4a70122b41a46f6b01e62b2c5ddace0d1ea92ca435ced0fda3f153d31fc70e25"}
Oct 30 22:39:59.832197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:59.832115    5503 generic.go:356] "Generic (PLEG): container finished" podID="6b6be266-a9f8-4938-8b03-1c258ebd83ce" containerID="a86ef63bd3878899e6ccc9268e685f53a897d24fa9f2c5c43ab12f90020023fb" exitCode=0
Oct 30 22:39:59.832197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:39:59.832161    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6" event={"ID":"6b6be266-a9f8-4938-8b03-1c258ebd83ce","Type":"ContainerDied","Data":"a86ef63bd3878899e6ccc9268e685f53a897d24fa9f2c5c43ab12f90020023fb"}
Oct 30 22:40:00.988028 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:00.988003    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:40:01.013226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.013199    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-openshift-ca\") pod \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") "
Oct 30 22:40:01.013361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.013255    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6be266-a9f8-4938-8b03-1c258ebd83ce-kube-api-access-72hr4\" (UniqueName: \"kubernetes.io/projected/6b6be266-a9f8-4938-8b03-1c258ebd83ce-kube-api-access-72hr4\") pod \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") "
Oct 30 22:40:01.013416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.013363    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-xks-ca\") pod \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\" (UID: \"6b6be266-a9f8-4938-8b03-1c258ebd83ce\") "
Oct 30 22:40:01.013546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.013518    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-openshift-ca" pod "6b6be266-a9f8-4938-8b03-1c258ebd83ce" (UID: "6b6be266-a9f8-4938-8b03-1c258ebd83ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:01.013731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.013714    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:01.015317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.015298    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-xks-ca" pod "6b6be266-a9f8-4938-8b03-1c258ebd83ce" (UID: "6b6be266-a9f8-4938-8b03-1c258ebd83ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:01.015392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.015351    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6be266-a9f8-4938-8b03-1c258ebd83ce-kube-api-access-72hr4" pod "6b6be266-a9f8-4938-8b03-1c258ebd83ce" (UID: "6b6be266-a9f8-4938-8b03-1c258ebd83ce"). InnerVolumeSpecName "kube-api-access-72hr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:01.114613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.114544    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6be266-a9f8-4938-8b03-1c258ebd83ce-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:01.114613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.114571    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72hr4\" (UniqueName: \"kubernetes.io/projected/6b6be266-a9f8-4938-8b03-1c258ebd83ce-kube-api-access-72hr4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:01.823859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.823831    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:01.845093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.845062    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"
Oct 30 22:40:01.845278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.845094    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6" event={"ID":"6b6be266-a9f8-4938-8b03-1c258ebd83ce","Type":"ContainerDied","Data":"4a70122b41a46f6b01e62b2c5ddace0d1ea92ca435ced0fda3f153d31fc70e25"}
Oct 30 22:40:01.845278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:01.845147    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a70122b41a46f6b01e62b2c5ddace0d1ea92ca435ced0fda3f153d31fc70e25"
Oct 30 22:40:05.680839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:05.680803    5503 scope.go:122] "RemoveContainer" containerID="50f4dc52a61de1d726f65d3ec83c23fa546d07f052459dea9ec7698829b0a395"
Oct 30 22:40:06.356578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:06.356542    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:10.921469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:10.921433    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:12.761494 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:12.761455    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:14.374452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:14.374418    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:40:15.544786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:15.544750    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:20.247773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:20.247738    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:24.797039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:24.797006    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:26.274810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:26.274777    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:26.437747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:26.437712    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:27.817360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:27.817328    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:28.098888 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:40:28.098801    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:40:29.069885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:29.069850    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:29.433736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:29.433706    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:34.034664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:34.034605    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:38.736941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:38.736904    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:43.224338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:43.224300    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:47.845585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:47.845551    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:52.107848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:52.107815    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:40:52.556857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:52.556820    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:57.220953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:40:57.220915    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:40:58.098917 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:40:58.098875    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:41:01.773288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:01.773245    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:03.913399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:03.913365    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:04.976010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:04.975959    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.171471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:05.171437    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:05.705507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:05.705474    5503 scope.go:122] "RemoveContainer" containerID="2d078af5a972a0a11593d215d75cb2f53b127074c4465486ce91e40b6aa7ba6e"
Oct 30 22:41:06.522114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:06.522081    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:09.771068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:09.771036    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:09.964348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:09.964315    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:10.145397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:10.145317    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:10.882392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:10.882347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:11.476585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:11.476553    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.171625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.171387    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.340285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.340253    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:13.677574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.676782    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"]
Oct 30 22:41:13.677574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.677390    5503 cpu_manager.go:414] "containerMap: removing container" podUID="6b6be266-a9f8-4938-8b03-1c258ebd83ce" containerName="revoke-keys"
Oct 30 22:41:13.677574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.677409    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6be266-a9f8-4938-8b03-1c258ebd83ce" containerName="revoke-keys"
Oct 30 22:41:13.677574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.677524    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6be266-a9f8-4938-8b03-1c258ebd83ce" containerName="revoke-keys"
Oct 30 22:41:13.682364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.682340    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:13.691010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.690977    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"]
Oct 30 22:41:13.736797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.736767    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-421286\""
Oct 30 22:41:13.737097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.737078    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-421286-dockercfg-55pf4\""
Oct 30 22:41:13.784120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.784080    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5930-5b0b-4e6a-8d0f-344044a3c904-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:13.784296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.784196    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2svd9\" (UniqueName: \"kubernetes.io/projected/a26b5930-5b0b-4e6a-8d0f-344044a3c904-kube-api-access-2svd9\") pod \"payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:13.885386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.885351    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2svd9\" (UniqueName: \"kubernetes.io/projected/a26b5930-5b0b-4e6a-8d0f-344044a3c904-kube-api-access-2svd9\") pod \"payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:13.885560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.885543    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5930-5b0b-4e6a-8d0f-344044a3c904-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:13.886250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.886223    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5930-5b0b-4e6a-8d0f-344044a3c904-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:13.948335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.948263    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2svd9\" (UniqueName: \"kubernetes.io/projected/a26b5930-5b0b-4e6a-8d0f-344044a3c904-kube-api-access-2svd9\") pod \"payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:13.998129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:13.998095    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:14.369274 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:41:14.369242    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26b5930_5b0b_4e6a_8d0f_344044a3c904.slice/crio-0bd0fb6a2e8232c2adff639f0c55f5d64a537b416e5a061c408bf182309c62d5 WatchSource:0}: Error finding container 0bd0fb6a2e8232c2adff639f0c55f5d64a537b416e5a061c408bf182309c62d5: Status 404 returned error can't find the container with id 0bd0fb6a2e8232c2adff639f0c55f5d64a537b416e5a061c408bf182309c62d5
Oct 30 22:41:14.371170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:14.371149    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"]
Oct 30 22:41:14.918359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:14.918323    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:15.292316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:15.292201    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t" event={"ID":"a26b5930-5b0b-4e6a-8d0f-344044a3c904","Type":"ContainerStarted","Data":"ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c"}
Oct 30 22:41:15.292316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:15.292308    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:15.292558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:15.292332    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t" event={"ID":"a26b5930-5b0b-4e6a-8d0f-344044a3c904","Type":"ContainerStarted","Data":"0bd0fb6a2e8232c2adff639f0c55f5d64a537b416e5a061c408bf182309c62d5"}
Oct 30 22:41:15.411459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:15.411408    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t" podStartSLOduration=2.411393535 podStartE2EDuration="2.411393535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:15.410695259 +0000 UTC m=+3251.745272027" watchObservedRunningTime="2026-10-30 22:41:15.411393535 +0000 UTC m=+3251.745970303"
Oct 30 22:41:16.338024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:16.337992    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:20.949907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:20.949824    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:22.911379 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.911348    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"]
Oct 30 22:41:22.915015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.914997    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:22.962396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.962364    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"]
Oct 30 22:41:22.962596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.962561    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy" containerID="cri-o://9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa" gracePeriod=30
Oct 30 22:41:22.969735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.969699    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:22.981080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.981043    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:22.981283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.981264    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:22.981432 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.981408    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sssn\" (UniqueName: \"kubernetes.io/projected/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-kube-api-access-2sssn\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:22.981573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.981555    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:22.981669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.981598    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:22.983665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:22.983646    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"]
Oct 30 22:41:23.070527 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.070500    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:41:23.083137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.083105    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:23.083270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.083155    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:23.083318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.083301    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sssn\" (UniqueName: \"kubernetes.io/projected/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-kube-api-access-2sssn\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:23.083862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.083839    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:23.085736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.085716    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:23.146052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.146018    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sssn\" (UniqueName: \"kubernetes.io/projected/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-kube-api-access-2sssn\") pod \"maas-api-revoke-sub-04cfdf32-j6vts\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:23.227655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.227543    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:23.604776 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:41:23.604729    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f03f99d_9ce5_4965_9737_eb50a5d14fbd.slice/crio-1bc432c5986c31aaacdc1e41f40bd31494f8bedae8e2f47a4a5e034cc3c342ba WatchSource:0}: Error finding container 1bc432c5986c31aaacdc1e41f40bd31494f8bedae8e2f47a4a5e034cc3c342ba: Status 404 returned error can't find the container with id 1bc432c5986c31aaacdc1e41f40bd31494f8bedae8e2f47a4a5e034cc3c342ba
Oct 30 22:41:23.607113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:23.607085    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"]
Oct 30 22:41:24.354260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:24.354179    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts" event={"ID":"6f03f99d-9ce5-4965-9737-eb50a5d14fbd","Type":"ContainerStarted","Data":"3ea70ffac6e609df4c7e0b435ceabab8aef0eef9d5c2fc5fcb531069043cba18"}
Oct 30 22:41:24.354260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:24.354231    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts" event={"ID":"6f03f99d-9ce5-4965-9737-eb50a5d14fbd","Type":"ContainerStarted","Data":"1bc432c5986c31aaacdc1e41f40bd31494f8bedae8e2f47a4a5e034cc3c342ba"}
Oct 30 22:41:24.474049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:24.473992    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts" podStartSLOduration=2.473979188 podStartE2EDuration="2.473979188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:24.47173441 +0000 UTC m=+3260.806311181" watchObservedRunningTime="2026-10-30 22:41:24.473979188 +0000 UTC m=+3260.808556000"
Oct 30 22:41:25.361521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:25.361481    5503 generic.go:356] "Generic (PLEG): container finished" podID="6f03f99d-9ce5-4965-9737-eb50a5d14fbd" containerID="3ea70ffac6e609df4c7e0b435ceabab8aef0eef9d5c2fc5fcb531069043cba18" exitCode=0
Oct 30 22:41:25.361957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:25.361520    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts" event={"ID":"6f03f99d-9ce5-4965-9737-eb50a5d14fbd","Type":"ContainerDied","Data":"3ea70ffac6e609df4c7e0b435ceabab8aef0eef9d5c2fc5fcb531069043cba18"}
Oct 30 22:41:25.610501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:25.610459    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:26.302474 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.302444    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:41:26.510651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.510595    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:26.587136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.587038    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:26.622888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.622856    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-xks-ca\") pod \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") "
Oct 30 22:41:26.623053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.622943    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-openshift-ca\") pod \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") "
Oct 30 22:41:26.623098 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.623049    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-kube-api-access-2sssn\" (UniqueName: \"kubernetes.io/projected/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-kube-api-access-2sssn\") pod \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\" (UID: \"6f03f99d-9ce5-4965-9737-eb50a5d14fbd\") "
Oct 30 22:41:26.623380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.623354    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-openshift-ca" pod "6f03f99d-9ce5-4965-9737-eb50a5d14fbd" (UID: "6f03f99d-9ce5-4965-9737-eb50a5d14fbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:26.623590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.623562    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:26.646589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.632023    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-xks-ca" pod "6f03f99d-9ce5-4965-9737-eb50a5d14fbd" (UID: "6f03f99d-9ce5-4965-9737-eb50a5d14fbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:26.660751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.651043    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-kube-api-access-2sssn" pod "6f03f99d-9ce5-4965-9737-eb50a5d14fbd" (UID: "6f03f99d-9ce5-4965-9737-eb50a5d14fbd"). InnerVolumeSpecName "kube-api-access-2sssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:26.724574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.724534    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:26.724574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:26.724575    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sssn\" (UniqueName: \"kubernetes.io/projected/6f03f99d-9ce5-4965-9737-eb50a5d14fbd-kube-api-access-2sssn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:27.374930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:27.374890    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"
Oct 30 22:41:27.374930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:27.374914    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts" event={"ID":"6f03f99d-9ce5-4965-9737-eb50a5d14fbd","Type":"ContainerDied","Data":"1bc432c5986c31aaacdc1e41f40bd31494f8bedae8e2f47a4a5e034cc3c342ba"}
Oct 30 22:41:27.375214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:27.374989    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc432c5986c31aaacdc1e41f40bd31494f8bedae8e2f47a4a5e034cc3c342ba"
Oct 30 22:41:28.102506 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:41:28.102469    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:41:28.160099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.160057    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:28.227716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.227691    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:41:28.349886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.349802    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-credential-socket\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.349886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.349850    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-envoy\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.349908    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-token\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-token\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.349938    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-data\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.349968    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-podinfo\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350009    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-certs\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350053    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-kube-api-access-n4jbr\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-kube-api-access-n4jbr\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350102    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-socket\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350106    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-credential-socket" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:28.350514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350238    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692a0bc0-19b2-42a2-b781-499bf4a62eef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692a0bc0-19b2-42a2-b781-499bf4a62eef-istiod-ca-cert\") pod \"692a0bc0-19b2-42a2-b781-499bf4a62eef\" (UID: \"692a0bc0-19b2-42a2-b781-499bf4a62eef\") "
Oct 30 22:41:28.350514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350338    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-data" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:28.350514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350403    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-socket" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:28.350716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350605    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-certs" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:28.350716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350702    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692a0bc0-19b2-42a2-b781-499bf4a62eef-istiod-ca-cert" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:28.350919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350897    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.350989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350924    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.350989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350942    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.350989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350957    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692a0bc0-19b2-42a2-b781-499bf4a62eef-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.350989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.350973    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.352251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.352221    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-podinfo" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:28.352373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.352252    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-envoy" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:28.352373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.352299    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-token" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:28.352373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.352328    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-kube-api-access-n4jbr" pod "692a0bc0-19b2-42a2-b781-499bf4a62eef" (UID: "692a0bc0-19b2-42a2-b781-499bf4a62eef"). InnerVolumeSpecName "kube-api-access-n4jbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:28.383591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.383554    5503 generic.go:356] "Generic (PLEG): container finished" podID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerID="9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa" exitCode=0
Oct 30 22:41:28.383785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.383619    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"
Oct 30 22:41:28.383785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.383660    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" event={"ID":"692a0bc0-19b2-42a2-b781-499bf4a62eef","Type":"ContainerDied","Data":"9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa"}
Oct 30 22:41:28.383785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.383719    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7" event={"ID":"692a0bc0-19b2-42a2-b781-499bf4a62eef","Type":"ContainerDied","Data":"2e8554b06afa4b78f9553e548f1ee566448bb266d9a708028c4079c395ce6e37"}
Oct 30 22:41:28.383785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.383743    5503 scope.go:122] "RemoveContainer" containerID="9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa"
Oct 30 22:41:28.395876 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.395853    5503 scope.go:122] "RemoveContainer" containerID="9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa"
Oct 30 22:41:28.396199 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:41:28.396176    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa\": container with ID starting with 9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa not found: ID does not exist" containerID="9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa"
Oct 30 22:41:28.396274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.396215    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa"} err="failed to get container status \"9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa\": rpc error: code = NotFound desc = could not find container \"9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa\": container with ID starting with 9616c8a050a4a1af80ec61581709dd26b22aab4e823285976c81ba411eb503fa not found: ID does not exist"
Oct 30 22:41:28.452184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.452143    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.452184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.452174    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.452184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.452187    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692a0bc0-19b2-42a2-b781-499bf4a62eef-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.452425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.452201    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jbr\" (UniqueName: \"kubernetes.io/projected/692a0bc0-19b2-42a2-b781-499bf4a62eef-kube-api-access-n4jbr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:28.559808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.559766    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"]
Oct 30 22:41:28.563232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:28.563204    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e524ce-openshift-default-85978f56c7-l94c7"]
Oct 30 22:41:29.378772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:29.378735    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:29.788437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:29.788402    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" path="/var/lib/kubelet/pods/692a0bc0-19b2-42a2-b781-499bf4a62eef/volumes"
Oct 30 22:41:30.351884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:30.351849    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:33.630650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.630614    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"]
Oct 30 22:41:33.631162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.631145    5503 cpu_manager.go:414] "containerMap: removing container" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy"
Oct 30 22:41:33.631209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.631166    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy"
Oct 30 22:41:33.631209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.631197    5503 cpu_manager.go:414] "containerMap: removing container" podUID="6f03f99d-9ce5-4965-9737-eb50a5d14fbd" containerName="revoke-keys"
Oct 30 22:41:33.631209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.631202    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f03f99d-9ce5-4965-9737-eb50a5d14fbd" containerName="revoke-keys"
Oct 30 22:41:33.631312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.631288    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="692a0bc0-19b2-42a2-b781-499bf4a62eef" containerName="istio-proxy"
Oct 30 22:41:33.631312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.631302    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f03f99d-9ce5-4965-9737-eb50a5d14fbd" containerName="revoke-keys"
Oct 30 22:41:33.640004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.639984    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.689270 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.689242    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"]
Oct 30 22:41:33.714108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.714068    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.714283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.714265    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghwj9\" (UniqueName: \"kubernetes.io/projected/716d904d-be7e-4045-b2da-33bb738089cd-kube-api-access-ghwj9\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.714342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.714311    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.737337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.737284    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:33.737337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.737286    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:33.737515 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.737352    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:41:33.744996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.744974    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:33.815868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.815833    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.816066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.815937    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwj9\" (UniqueName: \"kubernetes.io/projected/716d904d-be7e-4045-b2da-33bb738089cd-kube-api-access-ghwj9\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.816066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.815968    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.816531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.816506    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.818237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.818216    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.878560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.878534    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwj9\" (UniqueName: \"kubernetes.io/projected/716d904d-be7e-4045-b2da-33bb738089cd-kube-api-access-ghwj9\") pod \"maas-api-revoke-sub-e5da3c72-s6p4h\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:33.954895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:33.954868    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:34.331703 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:41:34.331671    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716d904d_be7e_4045_b2da_33bb738089cd.slice/crio-0bd69c1a32b4e76cc9ba47b6ad0e86386d40ec2e1e4664c96e7847150bd26f5d WatchSource:0}: Error finding container 0bd69c1a32b4e76cc9ba47b6ad0e86386d40ec2e1e4664c96e7847150bd26f5d: Status 404 returned error can't find the container with id 0bd69c1a32b4e76cc9ba47b6ad0e86386d40ec2e1e4664c96e7847150bd26f5d
Oct 30 22:41:34.333138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:34.333110    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"]
Oct 30 22:41:34.421862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:34.421829    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h" event={"ID":"716d904d-be7e-4045-b2da-33bb738089cd","Type":"ContainerStarted","Data":"0bd69c1a32b4e76cc9ba47b6ad0e86386d40ec2e1e4664c96e7847150bd26f5d"}
Oct 30 22:41:34.937673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:34.937618    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:35.429234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:35.429197    5503 generic.go:356] "Generic (PLEG): container finished" podID="716d904d-be7e-4045-b2da-33bb738089cd" containerID="7f09b0906f1369886e43f2153e5a7594bf314adaab246186cb9c3dc7f16a7c28" exitCode=0
Oct 30 22:41:35.429430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:35.429279    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h" event={"ID":"716d904d-be7e-4045-b2da-33bb738089cd","Type":"ContainerDied","Data":"7f09b0906f1369886e43f2153e5a7594bf314adaab246186cb9c3dc7f16a7c28"}
Oct 30 22:41:36.579456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.579433    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:36.646874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.646848    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716d904d-be7e-4045-b2da-33bb738089cd-kube-api-access-ghwj9\" (UniqueName: \"kubernetes.io/projected/716d904d-be7e-4045-b2da-33bb738089cd-kube-api-access-ghwj9\") pod \"716d904d-be7e-4045-b2da-33bb738089cd\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") "
Oct 30 22:41:36.647069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.646927    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-openshift-ca\") pod \"716d904d-be7e-4045-b2da-33bb738089cd\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") "
Oct 30 22:41:36.647069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.646955    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-xks-ca\") pod \"716d904d-be7e-4045-b2da-33bb738089cd\" (UID: \"716d904d-be7e-4045-b2da-33bb738089cd\") "
Oct 30 22:41:36.647317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.647290    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-openshift-ca" pod "716d904d-be7e-4045-b2da-33bb738089cd" (UID: "716d904d-be7e-4045-b2da-33bb738089cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:36.647478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.647455    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.649186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.649157    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716d904d-be7e-4045-b2da-33bb738089cd-kube-api-access-ghwj9" pod "716d904d-be7e-4045-b2da-33bb738089cd" (UID: "716d904d-be7e-4045-b2da-33bb738089cd"). InnerVolumeSpecName "kube-api-access-ghwj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:36.649186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.649164    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-xks-ca" pod "716d904d-be7e-4045-b2da-33bb738089cd" (UID: "716d904d-be7e-4045-b2da-33bb738089cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:36.748875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.748802    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghwj9\" (UniqueName: \"kubernetes.io/projected/716d904d-be7e-4045-b2da-33bb738089cd-kube-api-access-ghwj9\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.748875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:36.748828    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d904d-be7e-4045-b2da-33bb738089cd-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:37.442734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:37.442704    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"
Oct 30 22:41:37.442931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:37.442708    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h" event={"ID":"716d904d-be7e-4045-b2da-33bb738089cd","Type":"ContainerDied","Data":"0bd69c1a32b4e76cc9ba47b6ad0e86386d40ec2e1e4664c96e7847150bd26f5d"}
Oct 30 22:41:37.442931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:37.442828    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd69c1a32b4e76cc9ba47b6ad0e86386d40ec2e1e4664c96e7847150bd26f5d"
Oct 30 22:41:39.521079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:39.521035    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:44.026146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:44.026114    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:47.119452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:47.119415    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:48.615940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:48.615902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:52.988400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:52.988363    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:41:53.197689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:53.197648    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:53.534498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:53.534449    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:57.912345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:57.912312    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:41:58.098924 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:41:58.098890    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:41:59.105889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.105859    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"]
Oct 30 22:41:59.106355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.106339    5503 cpu_manager.go:414] "containerMap: removing container" podUID="716d904d-be7e-4045-b2da-33bb738089cd" containerName="revoke-keys"
Oct 30 22:41:59.106415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.106357    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716d904d-be7e-4045-b2da-33bb738089cd" containerName="revoke-keys"
Oct 30 22:41:59.106492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.106482    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="716d904d-be7e-4045-b2da-33bb738089cd" containerName="revoke-keys"
Oct 30 22:41:59.110004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.109988    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.165516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.165488    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"]
Oct 30 22:41:59.208364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.208327    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.208542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.208442    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndmr\" (UniqueName: \"kubernetes.io/projected/8aba1e21-48b5-40af-aeac-e19fb071ad27-kube-api-access-mndmr\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.208542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.208522    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.213537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.213516    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:41:59.213662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.213573    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:41:59.214541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.214524    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:59.214631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.214620    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:41:59.309948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.309916    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.310131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.309995    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndmr\" (UniqueName: \"kubernetes.io/projected/8aba1e21-48b5-40af-aeac-e19fb071ad27-kube-api-access-mndmr\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.310131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.310047    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.310668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.310610    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.312232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.312207    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.373130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.373058    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndmr\" (UniqueName: \"kubernetes.io/projected/8aba1e21-48b5-40af-aeac-e19fb071ad27-kube-api-access-mndmr\") pod \"maas-api-revoke-sub-58f22b95-vsjpp\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.422988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.422955    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:41:59.792952 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:41:59.792924    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aba1e21_48b5_40af_aeac_e19fb071ad27.slice/crio-df770055b9a1822f45a4a881b8c898b3f71ee8a92d1b7a3df84dbe245da8cec4 WatchSource:0}: Error finding container df770055b9a1822f45a4a881b8c898b3f71ee8a92d1b7a3df84dbe245da8cec4: Status 404 returned error can't find the container with id df770055b9a1822f45a4a881b8c898b3f71ee8a92d1b7a3df84dbe245da8cec4
Oct 30 22:41:59.795547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:41:59.795523    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"]
Oct 30 22:42:00.585693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:00.585657    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp" event={"ID":"8aba1e21-48b5-40af-aeac-e19fb071ad27","Type":"ContainerStarted","Data":"2666ad441e4d5b388dff0c968b9313011076ac3f4c5164bd4b1d9b7b11edbaf1"}
Oct 30 22:42:00.586199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:00.585721    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp" event={"ID":"8aba1e21-48b5-40af-aeac-e19fb071ad27","Type":"ContainerStarted","Data":"df770055b9a1822f45a4a881b8c898b3f71ee8a92d1b7a3df84dbe245da8cec4"}
Oct 30 22:42:00.703854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:00.703807    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp" podStartSLOduration=1.7037943389999999 podStartE2EDuration="1.703794339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:00.703435617 +0000 UTC m=+3297.038012403" watchObservedRunningTime="2026-10-30 22:42:00.703794339 +0000 UTC m=+3297.038371109"
Oct 30 22:42:01.593378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:01.593343    5503 generic.go:356] "Generic (PLEG): container finished" podID="8aba1e21-48b5-40af-aeac-e19fb071ad27" containerID="2666ad441e4d5b388dff0c968b9313011076ac3f4c5164bd4b1d9b7b11edbaf1" exitCode=0
Oct 30 22:42:01.593836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:01.593418    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp" event={"ID":"8aba1e21-48b5-40af-aeac-e19fb071ad27","Type":"ContainerDied","Data":"2666ad441e4d5b388dff0c968b9313011076ac3f4c5164bd4b1d9b7b11edbaf1"}
Oct 30 22:42:02.442521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.442488    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:02.750761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.750734    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:42:02.849052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.849019    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-openshift-ca\") pod \"8aba1e21-48b5-40af-aeac-e19fb071ad27\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") "
Oct 30 22:42:02.849231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.849087    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-xks-ca\") pod \"8aba1e21-48b5-40af-aeac-e19fb071ad27\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") "
Oct 30 22:42:02.849231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.849173    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aba1e21-48b5-40af-aeac-e19fb071ad27-kube-api-access-mndmr\" (UniqueName: \"kubernetes.io/projected/8aba1e21-48b5-40af-aeac-e19fb071ad27-kube-api-access-mndmr\") pod \"8aba1e21-48b5-40af-aeac-e19fb071ad27\" (UID: \"8aba1e21-48b5-40af-aeac-e19fb071ad27\") "
Oct 30 22:42:02.849385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.849358    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-openshift-ca" pod "8aba1e21-48b5-40af-aeac-e19fb071ad27" (UID: "8aba1e21-48b5-40af-aeac-e19fb071ad27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:02.849556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.849539    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:02.851202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.851178    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-xks-ca" pod "8aba1e21-48b5-40af-aeac-e19fb071ad27" (UID: "8aba1e21-48b5-40af-aeac-e19fb071ad27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:02.851310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.851268    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aba1e21-48b5-40af-aeac-e19fb071ad27-kube-api-access-mndmr" pod "8aba1e21-48b5-40af-aeac-e19fb071ad27" (UID: "8aba1e21-48b5-40af-aeac-e19fb071ad27"). InnerVolumeSpecName "kube-api-access-mndmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:02.950632 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.950602    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aba1e21-48b5-40af-aeac-e19fb071ad27-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:02.950632 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:02.950629    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndmr\" (UniqueName: \"kubernetes.io/projected/8aba1e21-48b5-40af-aeac-e19fb071ad27-kube-api-access-mndmr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:03.606564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:03.606522    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp" event={"ID":"8aba1e21-48b5-40af-aeac-e19fb071ad27","Type":"ContainerDied","Data":"df770055b9a1822f45a4a881b8c898b3f71ee8a92d1b7a3df84dbe245da8cec4"}
Oct 30 22:42:03.606564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:03.606558    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"
Oct 30 22:42:03.606811 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:03.606577    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df770055b9a1822f45a4a881b8c898b3f71ee8a92d1b7a3df84dbe245da8cec4"
Oct 30 22:42:04.227682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.227648    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"]
Oct 30 22:42:04.228304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.228282    5503 cpu_manager.go:414] "containerMap: removing container" podUID="8aba1e21-48b5-40af-aeac-e19fb071ad27" containerName="revoke-keys"
Oct 30 22:42:04.228412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.228305    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aba1e21-48b5-40af-aeac-e19fb071ad27" containerName="revoke-keys"
Oct 30 22:42:04.228537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.228521    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aba1e21-48b5-40af-aeac-e19fb071ad27" containerName="revoke-keys"
Oct 30 22:42:04.231683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.231660    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.266246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.266221    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.266390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.266280    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.266471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.266437    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmf75\" (UniqueName: \"kubernetes.io/projected/c477a1b8-b540-4d05-bc9f-f3c7110f9124-kube-api-access-gmf75\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.285965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.285888    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:04.286099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.286062    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:04.286210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.286190    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:04.336536 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.336502    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:42:04.336967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.336947    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:42:04.340009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.339984    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"]
Oct 30 22:42:04.345405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.345386    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:42:04.368033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.367994    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.368183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.368104    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.368259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.368192    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmf75\" (UniqueName: \"kubernetes.io/projected/c477a1b8-b540-4d05-bc9f-f3c7110f9124-kube-api-access-gmf75\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.368758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.368737    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.370559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.370535    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.436868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.436836    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmf75\" (UniqueName: \"kubernetes.io/projected/c477a1b8-b540-4d05-bc9f-f3c7110f9124-kube-api-access-gmf75\") pod \"maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.544322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.544248    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:04.916251 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:42:04.916214    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc477a1b8_b540_4d05_bc9f_f3c7110f9124.slice/crio-0fad8f1c4180e89b63afb525e42b942f55a0cf7ab620cd0919416d2f1d0c29d7 WatchSource:0}: Error finding container 0fad8f1c4180e89b63afb525e42b942f55a0cf7ab620cd0919416d2f1d0c29d7: Status 404 returned error can't find the container with id 0fad8f1c4180e89b63afb525e42b942f55a0cf7ab620cd0919416d2f1d0c29d7
Oct 30 22:42:04.918774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:04.918748    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"]
Oct 30 22:42:05.628063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:05.627964    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5" event={"ID":"c477a1b8-b540-4d05-bc9f-f3c7110f9124","Type":"ContainerStarted","Data":"8fa4c40f969f2355e8e72562bf13dd2836bdbdd12deb86939a41f99cd54b8c55"}
Oct 30 22:42:05.628063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:05.628023    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5" event={"ID":"c477a1b8-b540-4d05-bc9f-f3c7110f9124","Type":"ContainerStarted","Data":"0fad8f1c4180e89b63afb525e42b942f55a0cf7ab620cd0919416d2f1d0c29d7"}
Oct 30 22:42:05.747978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:05.747934    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5" podStartSLOduration=1.747921675 podStartE2EDuration="1.747921675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:05.747355409 +0000 UTC m=+3302.081932178" watchObservedRunningTime="2026-10-30 22:42:05.747921675 +0000 UTC m=+3302.082498442"
Oct 30 22:42:06.635440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:06.635398    5503 generic.go:356] "Generic (PLEG): container finished" podID="c477a1b8-b540-4d05-bc9f-f3c7110f9124" containerID="8fa4c40f969f2355e8e72562bf13dd2836bdbdd12deb86939a41f99cd54b8c55" exitCode=0
Oct 30 22:42:06.635967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:06.635478    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5" event={"ID":"c477a1b8-b540-4d05-bc9f-f3c7110f9124","Type":"ContainerDied","Data":"8fa4c40f969f2355e8e72562bf13dd2836bdbdd12deb86939a41f99cd54b8c55"}
Oct 30 22:42:07.453252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.453216    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:07.792761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.792736    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:07.909464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.909431    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-xks-ca\") pod \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") "
Oct 30 22:42:07.909674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.909512    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-openshift-ca\") pod \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") "
Oct 30 22:42:07.909772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.909683    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c477a1b8-b540-4d05-bc9f-f3c7110f9124-kube-api-access-gmf75\" (UniqueName: \"kubernetes.io/projected/c477a1b8-b540-4d05-bc9f-f3c7110f9124-kube-api-access-gmf75\") pod \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\" (UID: \"c477a1b8-b540-4d05-bc9f-f3c7110f9124\") "
Oct 30 22:42:07.909898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.909863    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-openshift-ca" pod "c477a1b8-b540-4d05-bc9f-f3c7110f9124" (UID: "c477a1b8-b540-4d05-bc9f-f3c7110f9124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:07.909898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.910139    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:07.911949 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.911923    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-xks-ca" pod "c477a1b8-b540-4d05-bc9f-f3c7110f9124" (UID: "c477a1b8-b540-4d05-bc9f-f3c7110f9124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:07.912261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:07.912240    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c477a1b8-b540-4d05-bc9f-f3c7110f9124-kube-api-access-gmf75" pod "c477a1b8-b540-4d05-bc9f-f3c7110f9124" (UID: "c477a1b8-b540-4d05-bc9f-f3c7110f9124"). InnerVolumeSpecName "kube-api-access-gmf75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:08.011581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:08.011549    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c477a1b8-b540-4d05-bc9f-f3c7110f9124-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:08.011581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:08.011577    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmf75\" (UniqueName: \"kubernetes.io/projected/c477a1b8-b540-4d05-bc9f-f3c7110f9124-kube-api-access-gmf75\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:08.648551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:08.648522    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"
Oct 30 22:42:08.648551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:08.648545    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5" event={"ID":"c477a1b8-b540-4d05-bc9f-f3c7110f9124","Type":"ContainerDied","Data":"0fad8f1c4180e89b63afb525e42b942f55a0cf7ab620cd0919416d2f1d0c29d7"}
Oct 30 22:42:08.648851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:08.648661    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fad8f1c4180e89b63afb525e42b942f55a0cf7ab620cd0919416d2f1d0c29d7"
Oct 30 22:42:09.820575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.820544    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"]
Oct 30 22:42:09.821097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.821027    5503 cpu_manager.go:414] "containerMap: removing container" podUID="c477a1b8-b540-4d05-bc9f-f3c7110f9124" containerName="revoke-keys"
Oct 30 22:42:09.821097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.821040    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c477a1b8-b540-4d05-bc9f-f3c7110f9124" containerName="revoke-keys"
Oct 30 22:42:09.821244 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.821141    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c477a1b8-b540-4d05-bc9f-f3c7110f9124" containerName="revoke-keys"
Oct 30 22:42:09.824504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.824484    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:09.879897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.879858    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"]
Oct 30 22:42:09.928181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.928135    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:42:09.928181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.928141    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:09.928181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.928185    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:42:09.930605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.930590    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:42:09.932186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.932168    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:09.932246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.932209    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhg8m\" (UniqueName: \"kubernetes.io/projected/2dd52f46-7694-4088-8ba1-5a2ae3167191-kube-api-access-lhg8m\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:09.932291 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:09.932239    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.033210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.033177    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.033420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.033224    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhg8m\" (UniqueName: \"kubernetes.io/projected/2dd52f46-7694-4088-8ba1-5a2ae3167191-kube-api-access-lhg8m\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.033420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.033252    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.033952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.033928    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.035501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.035480    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.095778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.095713    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhg8m\" (UniqueName: \"kubernetes.io/projected/2dd52f46-7694-4088-8ba1-5a2ae3167191-kube-api-access-lhg8m\") pod \"maas-api-revoke-sub-6b2e95ae-f6hvk\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.139109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.139080    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:10.512325 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:42:10.512289    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd52f46_7694_4088_8ba1_5a2ae3167191.slice/crio-848b0bac8c8dc793af10df301bfe70cf510439464888d990bd2755b9e6b8c20c WatchSource:0}: Error finding container 848b0bac8c8dc793af10df301bfe70cf510439464888d990bd2755b9e6b8c20c: Status 404 returned error can't find the container with id 848b0bac8c8dc793af10df301bfe70cf510439464888d990bd2755b9e6b8c20c
Oct 30 22:42:10.514623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.514599    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"]
Oct 30 22:42:10.663258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:10.663223    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk" event={"ID":"2dd52f46-7694-4088-8ba1-5a2ae3167191","Type":"ContainerStarted","Data":"848b0bac8c8dc793af10df301bfe70cf510439464888d990bd2755b9e6b8c20c"}
Oct 30 22:42:11.670835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:11.670799    5503 generic.go:356] "Generic (PLEG): container finished" podID="2dd52f46-7694-4088-8ba1-5a2ae3167191" containerID="d28fadb9a686d6dae223914eacc60c42287ea2a2c0e7ede3cf371e4b11b80d77" exitCode=0
Oct 30 22:42:11.671222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:11.670887    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk" event={"ID":"2dd52f46-7694-4088-8ba1-5a2ae3167191","Type":"ContainerDied","Data":"d28fadb9a686d6dae223914eacc60c42287ea2a2c0e7ede3cf371e4b11b80d77"}
Oct 30 22:42:12.042856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.042818    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:12.827947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.827922    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:12.963485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.963408    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd52f46-7694-4088-8ba1-5a2ae3167191-kube-api-access-lhg8m\" (UniqueName: \"kubernetes.io/projected/2dd52f46-7694-4088-8ba1-5a2ae3167191-kube-api-access-lhg8m\") pod \"2dd52f46-7694-4088-8ba1-5a2ae3167191\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") "
Oct 30 22:42:12.963658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.963619    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-xks-ca\") pod \"2dd52f46-7694-4088-8ba1-5a2ae3167191\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") "
Oct 30 22:42:12.963733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.963686    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-openshift-ca\") pod \"2dd52f46-7694-4088-8ba1-5a2ae3167191\" (UID: \"2dd52f46-7694-4088-8ba1-5a2ae3167191\") "
Oct 30 22:42:12.964099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.964069    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-openshift-ca" pod "2dd52f46-7694-4088-8ba1-5a2ae3167191" (UID: "2dd52f46-7694-4088-8ba1-5a2ae3167191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:12.965714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.965682    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd52f46-7694-4088-8ba1-5a2ae3167191-kube-api-access-lhg8m" pod "2dd52f46-7694-4088-8ba1-5a2ae3167191" (UID: "2dd52f46-7694-4088-8ba1-5a2ae3167191"). InnerVolumeSpecName "kube-api-access-lhg8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:12.965829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:12.965791    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-xks-ca" pod "2dd52f46-7694-4088-8ba1-5a2ae3167191" (UID: "2dd52f46-7694-4088-8ba1-5a2ae3167191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:13.065467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:13.065428    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:13.065467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:13.065462    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd52f46-7694-4088-8ba1-5a2ae3167191-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:13.065467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:13.065472    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhg8m\" (UniqueName: \"kubernetes.io/projected/2dd52f46-7694-4088-8ba1-5a2ae3167191-kube-api-access-lhg8m\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:13.684132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:13.684094    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk" event={"ID":"2dd52f46-7694-4088-8ba1-5a2ae3167191","Type":"ContainerDied","Data":"848b0bac8c8dc793af10df301bfe70cf510439464888d990bd2755b9e6b8c20c"}
Oct 30 22:42:13.684132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:13.684110    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"
Oct 30 22:42:13.684396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:13.684166    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="848b0bac8c8dc793af10df301bfe70cf510439464888d990bd2755b9e6b8c20c"
Oct 30 22:42:15.797155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:15.797124    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"]
Oct 30 22:42:15.906208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:15.906162    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"]
Oct 30 22:42:16.187382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:16.187349    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:16.336807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:16.336772    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:16.733539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:16.733504    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:20.798247 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:20.798200    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh" podUID="405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" containerName="payload-processing" containerID="cri-o://57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58" gracePeriod=25
Oct 30 22:42:20.907877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:20.907829    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw" podUID="184102c8-ecdc-4c5e-b07d-868cbcd4324d" containerName="payload-pre-processing" containerID="cri-o://a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0" gracePeriod=25
Oct 30 22:42:21.061944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.061879    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:42:21.164560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.164528    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-plugins-config-volume\") pod \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") "
Oct 30 22:42:21.164771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.164756    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-kube-api-access-qlh7d\" (UniqueName: \"kubernetes.io/projected/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-kube-api-access-qlh7d\") pod \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\" (UID: \"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b\") "
Oct 30 22:42:21.164909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.164883    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-plugins-config-volume" pod "405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" (UID: "405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:21.165151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.165133    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:21.166736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.166716    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-kube-api-access-qlh7d" pod "405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" (UID: "405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b"). InnerVolumeSpecName "kube-api-access-qlh7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:21.184588 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.184568    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:42:21.266585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.266548    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184102c8-ecdc-4c5e-b07d-868cbcd4324d-kube-api-access-48r59\" (UniqueName: \"kubernetes.io/projected/184102c8-ecdc-4c5e-b07d-868cbcd4324d-kube-api-access-48r59\") pod \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") "
Oct 30 22:42:21.266850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.266827    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184102c8-ecdc-4c5e-b07d-868cbcd4324d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184102c8-ecdc-4c5e-b07d-868cbcd4324d-plugins-config-volume\") pod \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\" (UID: \"184102c8-ecdc-4c5e-b07d-868cbcd4324d\") "
Oct 30 22:42:21.267173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.267150    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184102c8-ecdc-4c5e-b07d-868cbcd4324d-plugins-config-volume" pod "184102c8-ecdc-4c5e-b07d-868cbcd4324d" (UID: "184102c8-ecdc-4c5e-b07d-868cbcd4324d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:21.267259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.267246    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlh7d\" (UniqueName: \"kubernetes.io/projected/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b-kube-api-access-qlh7d\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:21.267297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.267264    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/184102c8-ecdc-4c5e-b07d-868cbcd4324d-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:21.268626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.268605    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184102c8-ecdc-4c5e-b07d-868cbcd4324d-kube-api-access-48r59" pod "184102c8-ecdc-4c5e-b07d-868cbcd4324d" (UID: "184102c8-ecdc-4c5e-b07d-868cbcd4324d"). InnerVolumeSpecName "kube-api-access-48r59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:21.279972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.279946    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:21.368556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.368480    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48r59\" (UniqueName: \"kubernetes.io/projected/184102c8-ecdc-4c5e-b07d-868cbcd4324d-kube-api-access-48r59\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:21.472737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.472697    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:21.631609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.631525    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:21.743253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.743217    5503 generic.go:356] "Generic (PLEG): container finished" podID="184102c8-ecdc-4c5e-b07d-868cbcd4324d" containerID="a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0" exitCode=0
Oct 30 22:42:21.743454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.743278    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"
Oct 30 22:42:21.743454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.743297    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw" event={"ID":"184102c8-ecdc-4c5e-b07d-868cbcd4324d","Type":"ContainerDied","Data":"a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0"}
Oct 30 22:42:21.743454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.743388    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw" event={"ID":"184102c8-ecdc-4c5e-b07d-868cbcd4324d","Type":"ContainerDied","Data":"1c7f615f68ac22deb1f5bbd8bf8a655d6f3e3f182914ad71422a02ba2e954ed6"}
Oct 30 22:42:21.743454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.743424    5503 scope.go:122] "RemoveContainer" containerID="a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0"
Oct 30 22:42:21.744826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.744801    5503 generic.go:356] "Generic (PLEG): container finished" podID="405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" containerID="57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58" exitCode=0
Oct 30 22:42:21.744950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.744885    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"
Oct 30 22:42:21.744950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.744901    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh" event={"ID":"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b","Type":"ContainerDied","Data":"57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58"}
Oct 30 22:42:21.744950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.744937    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh" event={"ID":"405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b","Type":"ContainerDied","Data":"d9d554f9f4e190b38d10b3fe85d2f3c5214b9bdfddbbe5bdfc1acb977ae8b228"}
Oct 30 22:42:21.754188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.754172    5503 scope.go:122] "RemoveContainer" containerID="a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0"
Oct 30 22:42:21.754579 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:42:21.754442    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0\": container with ID starting with a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0 not found: ID does not exist" containerID="a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0"
Oct 30 22:42:21.754579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.754470    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0"} err="failed to get container status \"a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0\": rpc error: code = NotFound desc = could not find container \"a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0\": container with ID starting with a2f2ad183cedc44b656636712de9c91848a05cc4896929251697e8034d4d5eb0 not found: ID does not exist"
Oct 30 22:42:21.754579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.754492    5503 scope.go:122] "RemoveContainer" containerID="57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58"
Oct 30 22:42:21.763357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.763333    5503 scope.go:122] "RemoveContainer" containerID="57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58"
Oct 30 22:42:21.763646 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:42:21.763620    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58\": container with ID starting with 57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58 not found: ID does not exist" containerID="57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58"
Oct 30 22:42:21.763699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.763656    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58"} err="failed to get container status \"57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58\": rpc error: code = NotFound desc = could not find container \"57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58\": container with ID starting with 57f7d6ce6269f71640a7bf2e6751b5aff662e439310021b3986b1b8b057bfe58 not found: ID does not exist"
Oct 30 22:42:21.912237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.912209    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"]
Oct 30 22:42:21.916897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:21.916875    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-cbe3a9-c7485d8c-xz9bh"]
Oct 30 22:42:22.029882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:22.029845    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"]
Oct 30 22:42:22.034126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:22.034106    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cbe3a9-8495794667-xlqrw"]
Oct 30 22:42:23.789068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:23.789036    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184102c8-ecdc-4c5e-b07d-868cbcd4324d" path="/var/lib/kubelet/pods/184102c8-ecdc-4c5e-b07d-868cbcd4324d/volumes"
Oct 30 22:42:23.789452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:23.789433    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" path="/var/lib/kubelet/pods/405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b/volumes"
Oct 30 22:42:25.313184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:25.313139    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:25.893230 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:25.893192    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:26.182573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:26.182538    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:26.339932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:26.339900    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:28.099190 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:42:28.099153    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:42:28.497234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:28.497188    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:28.667588 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:28.667495    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:29.771835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:29.771806    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:30.120919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:30.120807    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:30.296266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:30.296230    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:30.459060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:30.459028    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:30.898844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:30.898734    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:33.450576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.450509    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:33.652182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.652141    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:33.875280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875202    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"]
Oct 30 22:42:33.875715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875701    5503 cpu_manager.go:414] "containerMap: removing container" podUID="405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" containerName="payload-processing"
Oct 30 22:42:33.875768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875718    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" containerName="payload-processing"
Oct 30 22:42:33.875768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875752    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2dd52f46-7694-4088-8ba1-5a2ae3167191" containerName="revoke-keys"
Oct 30 22:42:33.875768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875758    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd52f46-7694-4088-8ba1-5a2ae3167191" containerName="revoke-keys"
Oct 30 22:42:33.875867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875778    5503 cpu_manager.go:414] "containerMap: removing container" podUID="184102c8-ecdc-4c5e-b07d-868cbcd4324d" containerName="payload-pre-processing"
Oct 30 22:42:33.875867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875783    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184102c8-ecdc-4c5e-b07d-868cbcd4324d" containerName="payload-pre-processing"
Oct 30 22:42:33.875932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875881    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="405d6bc7-9fbd-4c48-add2-5fa35e9bdf4b" containerName="payload-processing"
Oct 30 22:42:33.875932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875894    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd52f46-7694-4088-8ba1-5a2ae3167191" containerName="revoke-keys"
Oct 30 22:42:33.875932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.875904    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="184102c8-ecdc-4c5e-b07d-868cbcd4324d" containerName="payload-pre-processing"
Oct 30 22:42:33.880532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.880509    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:33.926316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.926280    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lnr\" (UniqueName: \"kubernetes.io/projected/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-kube-api-access-l4lnr\") pod \"payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:33.926504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.926415    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:33.935583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.935540    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"]
Oct 30 22:42:33.982664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.982584    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b829bc-dockercfg-bljc2\""
Oct 30 22:42:33.982664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:33.982609    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b829bc\""
Oct 30 22:42:34.028220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.028181    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:34.028452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.028297    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lnr\" (UniqueName: \"kubernetes.io/projected/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-kube-api-access-l4lnr\") pod \"payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:34.028982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.028960    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:34.036000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.035973    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"]
Oct 30 22:42:34.040032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.040013    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.079281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.079253    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"]
Oct 30 22:42:34.091190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.091156    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lnr\" (UniqueName: \"kubernetes.io/projected/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-kube-api-access-l4lnr\") pod \"payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:34.129936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.129859    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df960647-587c-4ce6-869f-9eecd3c412b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.130088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.129973    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2gk8\" (UniqueName: \"kubernetes.io/projected/df960647-587c-4ce6-869f-9eecd3c412b9-kube-api-access-k2gk8\") pod \"payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.194251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.194215    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:34.231246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.231213    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df960647-587c-4ce6-869f-9eecd3c412b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.231403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.231334    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2gk8\" (UniqueName: \"kubernetes.io/projected/df960647-587c-4ce6-869f-9eecd3c412b9-kube-api-access-k2gk8\") pod \"payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.232003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.231984    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df960647-587c-4ce6-869f-9eecd3c412b9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.293891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.293860    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2gk8\" (UniqueName: \"kubernetes.io/projected/df960647-587c-4ce6-869f-9eecd3c412b9-kube-api-access-k2gk8\") pod \"payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.354578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.354550    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.566934 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:42:34.566903    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8ec4e3_61c4_4e4c_a88e_223288db0cb3.slice/crio-96824ff588ccc2c3d9870b2c56ebc6a7c5eb7487bec2ca9b101797e5a0b31b9f WatchSource:0}: Error finding container 96824ff588ccc2c3d9870b2c56ebc6a7c5eb7487bec2ca9b101797e5a0b31b9f: Status 404 returned error can't find the container with id 96824ff588ccc2c3d9870b2c56ebc6a7c5eb7487bec2ca9b101797e5a0b31b9f
Oct 30 22:42:34.570523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.570501    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"]
Oct 30 22:42:34.723211 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:42:34.723177    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf960647_587c_4ce6_869f_9eecd3c412b9.slice/crio-d08bf88ac208842827fe35c999d0ff8368964df9d80b5cf8db1e76aa00bff501 WatchSource:0}: Error finding container d08bf88ac208842827fe35c999d0ff8368964df9d80b5cf8db1e76aa00bff501: Status 404 returned error can't find the container with id d08bf88ac208842827fe35c999d0ff8368964df9d80b5cf8db1e76aa00bff501
Oct 30 22:42:34.726044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.726016    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"]
Oct 30 22:42:34.827543 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.827510    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb" event={"ID":"df960647-587c-4ce6-869f-9eecd3c412b9","Type":"ContainerStarted","Data":"dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9"}
Oct 30 22:42:34.827719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.827600    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb" event={"ID":"df960647-587c-4ce6-869f-9eecd3c412b9","Type":"ContainerStarted","Data":"d08bf88ac208842827fe35c999d0ff8368964df9d80b5cf8db1e76aa00bff501"}
Oct 30 22:42:34.827813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.827743    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:34.829382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.829357    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg" event={"ID":"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3","Type":"ContainerStarted","Data":"0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d"}
Oct 30 22:42:34.829486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.829401    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg" event={"ID":"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3","Type":"ContainerStarted","Data":"96824ff588ccc2c3d9870b2c56ebc6a7c5eb7487bec2ca9b101797e5a0b31b9f"}
Oct 30 22:42:34.829486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.829446    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:34.950160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:34.950107    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb" podStartSLOduration=1.950092521 podStartE2EDuration="1.950092521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:34.946021354 +0000 UTC m=+3331.280598125" watchObservedRunningTime="2026-10-30 22:42:34.950092521 +0000 UTC m=+3331.284669290"
Oct 30 22:42:35.082744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:35.081218    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg" podStartSLOduration=2.081198393 podStartE2EDuration="2.081198393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:35.064063799 +0000 UTC m=+3331.398640597" watchObservedRunningTime="2026-10-30 22:42:35.081198393 +0000 UTC m=+3331.415775164"
Oct 30 22:42:36.108609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:36.108573    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:38.826061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:38.826030    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:42:41.183911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:41.183881    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:45.768135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:45.768095    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:45.851052 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:45.851018    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:42:45.852475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:45.852454    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:42:46.356341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:46.356299    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:50.727464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.727428    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"]
Oct 30 22:42:50.730667 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.730627    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.784755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.784729    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-421286-openshift-default-dockercfg-qlszb\""
Oct 30 22:42:50.813138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813104    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813160    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvtx5\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-kube-api-access-pvtx5\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813227    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813319    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813364    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813392    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813424    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813500    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.813605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.813556    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.839961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.839929    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"]
Oct 30 22:42:50.914596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.914559    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.914805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.914664    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvtx5\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-kube-api-access-pvtx5\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.914805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.914721    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.914805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.914759    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.914979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.914924    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.914983    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915018    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915062    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915166    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915217    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915267    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915391    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915447    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.915948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.915921    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.917084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.917051    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.917748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.917722    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.923584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.923556    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:50.980342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.980262    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:50.980518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:50.980380    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvtx5\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-kube-api-access-pvtx5\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:51.043361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:51.043328    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:51.463064 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:42:51.463030    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df42c1d_bc6f_47f1_a75c_7b98f2fca6ef.slice/crio-f340609bdbe0df2d9f8fbb4e5d1901f9f53be60a8370245455a02e065a78698a WatchSource:0}: Error finding container f340609bdbe0df2d9f8fbb4e5d1901f9f53be60a8370245455a02e065a78698a: Status 404 returned error can't find the container with id f340609bdbe0df2d9f8fbb4e5d1901f9f53be60a8370245455a02e065a78698a
Oct 30 22:42:51.465505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:51.465482    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"]
Oct 30 22:42:51.465776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:51.465745    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:42:51.465829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:51.465808    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:42:51.465866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:51.465837    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:42:51.944656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:51.944608    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" event={"ID":"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef","Type":"ContainerStarted","Data":"8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd"}
Oct 30 22:42:51.945058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:51.944684    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" event={"ID":"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef","Type":"ContainerStarted","Data":"f340609bdbe0df2d9f8fbb4e5d1901f9f53be60a8370245455a02e065a78698a"}
Oct 30 22:42:52.044029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:52.043995    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:52.045676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:52.045628    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:42:52.045805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:52.045696    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 22:42:52.073816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:52.073761    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podStartSLOduration=2.073746823 podStartE2EDuration="2.073746823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:52.07279059 +0000 UTC m=+3348.407367361" watchObservedRunningTime="2026-10-30 22:42:52.073746823 +0000 UTC m=+3348.408323658"
Oct 30 22:42:53.044192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:53.044150    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:42:53.044591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:53.044219    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 22:42:54.044654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:54.044592    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:42:54.045046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:54.044704    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 22:42:55.044043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:55.044002    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:42:55.044232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:55.044062    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 22:42:55.771436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:55.771399    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:42:56.044565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:56.044471    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:42:56.044565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:56.044541    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 22:42:57.044680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:57.044612    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Oct 30 22:42:57.045195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:57.044718    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Oct 30 22:42:58.048131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:58.048101    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:58.048588 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:58.048557    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:58.049314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:42:58.049294    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:42:58.098577 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:42:58.098550    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:43:00.487841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:00.487309    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:02.844867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:02.844827    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"]
Oct 30 22:43:02.882921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:02.882886    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:05.139357 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:05.139307    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:05.744789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:05.744758    5503 scope.go:122] "RemoveContainer" containerID="4765d94af54a1722286b3bf095b565027017a9add8ff7d1ea528bca833f3fd59"
Oct 30 22:43:09.730305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:09.730272    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:14.279542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:14.279504    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:17.867528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:17.867484    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv" podUID="f9bf1218-0919-4fda-80b5-c206c80bc169" containerName="main" containerID="cri-o://04af6fad708c97226087ce59a76af71cdfaed8c2b02eab9de6a471889754457b" gracePeriod=45
Oct 30 22:43:18.107184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.107154    5503 generic.go:356] "Generic (PLEG): container finished" podID="f9bf1218-0919-4fda-80b5-c206c80bc169" containerID="04af6fad708c97226087ce59a76af71cdfaed8c2b02eab9de6a471889754457b" exitCode=0
Oct 30 22:43:18.107312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.107226    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv" event={"ID":"f9bf1218-0919-4fda-80b5-c206c80bc169","Type":"ContainerDied","Data":"04af6fad708c97226087ce59a76af71cdfaed8c2b02eab9de6a471889754457b"}
Oct 30 22:43:18.107312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.107271    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv" event={"ID":"f9bf1218-0919-4fda-80b5-c206c80bc169","Type":"ContainerDied","Data":"aa4442ac79231e62b8e663cca4f8cd86e621d1be10f31bf42b114bd49f252cf4"}
Oct 30 22:43:18.107312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.107286    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4442ac79231e62b8e663cca4f8cd86e621d1be10f31bf42b114bd49f252cf4"
Oct 30 22:43:18.122348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.122293    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:43:18.251467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.251437    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bf1218-0919-4fda-80b5-c206c80bc169-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bf1218-0919-4fda-80b5-c206c80bc169-tls-certs\") pod \"f9bf1218-0919-4fda-80b5-c206c80bc169\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") "
Oct 30 22:43:18.251666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.251550    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-home\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-home\") pod \"f9bf1218-0919-4fda-80b5-c206c80bc169\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") "
Oct 30 22:43:18.251666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.251589    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-model-cache\") pod \"f9bf1218-0919-4fda-80b5-c206c80bc169\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") "
Oct 30 22:43:18.251816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.251683    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-dshm\") pod \"f9bf1218-0919-4fda-80b5-c206c80bc169\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") "
Oct 30 22:43:18.251816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.251779    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bf1218-0919-4fda-80b5-c206c80bc169-kube-api-access-hss9g\" (UniqueName: \"kubernetes.io/projected/f9bf1218-0919-4fda-80b5-c206c80bc169-kube-api-access-hss9g\") pod \"f9bf1218-0919-4fda-80b5-c206c80bc169\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") "
Oct 30 22:43:18.251918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.251822    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-tmp-dir\") pod \"f9bf1218-0919-4fda-80b5-c206c80bc169\" (UID: \"f9bf1218-0919-4fda-80b5-c206c80bc169\") "
Oct 30 22:43:18.251918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.251860    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-model-cache" pod "f9bf1218-0919-4fda-80b5-c206c80bc169" (UID: "f9bf1218-0919-4fda-80b5-c206c80bc169"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:18.252144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.252074    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-home" pod "f9bf1218-0919-4fda-80b5-c206c80bc169" (UID: "f9bf1218-0919-4fda-80b5-c206c80bc169"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:18.252237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.252149    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-tmp-dir" pod "f9bf1218-0919-4fda-80b5-c206c80bc169" (UID: "f9bf1218-0919-4fda-80b5-c206c80bc169"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:18.252378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.252357    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:18.252461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.252383    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:18.252461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.252397    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:18.254072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.254044    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bf1218-0919-4fda-80b5-c206c80bc169-tls-certs" pod "f9bf1218-0919-4fda-80b5-c206c80bc169" (UID: "f9bf1218-0919-4fda-80b5-c206c80bc169"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:18.254338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.254309    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-dshm" pod "f9bf1218-0919-4fda-80b5-c206c80bc169" (UID: "f9bf1218-0919-4fda-80b5-c206c80bc169"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:18.254442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.254349    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bf1218-0919-4fda-80b5-c206c80bc169-kube-api-access-hss9g" pod "f9bf1218-0919-4fda-80b5-c206c80bc169" (UID: "f9bf1218-0919-4fda-80b5-c206c80bc169"). InnerVolumeSpecName "kube-api-access-hss9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:18.353531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.353503    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bf1218-0919-4fda-80b5-c206c80bc169-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:18.353531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.353528    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bf1218-0919-4fda-80b5-c206c80bc169-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:18.353748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.353540    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hss9g\" (UniqueName: \"kubernetes.io/projected/f9bf1218-0919-4fda-80b5-c206c80bc169-kube-api-access-hss9g\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:18.556546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.556513    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:43:18.752219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:18.752190    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:19.131308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:19.131270    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"
Oct 30 22:43:19.303325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:19.303289    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"]
Oct 30 22:43:19.308231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:19.308201    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc6466/auth-test-model-fc6466-kserve-76bb69cd97-l66qv"]
Oct 30 22:43:19.789422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:19.789392    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bf1218-0919-4fda-80b5-c206c80bc169" path="/var/lib/kubelet/pods/f9bf1218-0919-4fda-80b5-c206c80bc169/volumes"
Oct 30 22:43:23.488470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:23.488427    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:24.628584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:24.628551    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:25.165698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.165664    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"]
Oct 30 22:43:25.166188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.166173    5503 cpu_manager.go:414] "containerMap: removing container" podUID="f9bf1218-0919-4fda-80b5-c206c80bc169" containerName="main"
Oct 30 22:43:25.166188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.166187    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bf1218-0919-4fda-80b5-c206c80bc169" containerName="main"
Oct 30 22:43:25.166363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.166299    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bf1218-0919-4fda-80b5-c206c80bc169" containerName="main"
Oct 30 22:43:25.170866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.170849    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.232042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.232003    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9a60dd-openshift-default-dockercfg-wxdch\""
Oct 30 22:43:25.281578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.281517    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:25.292906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.292869    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"]
Oct 30 22:43:25.362545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.362502    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-envoy\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.362845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.362812    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-data\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.363008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.362991    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-podinfo\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.369773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.369742    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-token\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.369973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.369956    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c163a492-0c0a-4664-805b-5798e4cd42a2-istiod-ca-cert\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.371790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.371760    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-socket\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.372072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.372051    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-certs\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.372273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.372255    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6zh\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-kube-api-access-ts6zh\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.383053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.383026    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-credential-socket\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484261    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-credential-socket\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484348    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-envoy\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484378    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-data\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484397    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-podinfo\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484429    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-token\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484562    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c163a492-0c0a-4664-805b-5798e4cd42a2-istiod-ca-cert\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484676    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-socket\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484716    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-certs\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484740    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-credential-socket\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.484822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484763    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6zh\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-kube-api-access-ts6zh\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.485137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.484821    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-data\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.485137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.485015    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-socket\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.485137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.485067    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-certs\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.485333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.485311    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c163a492-0c0a-4664-805b-5798e4cd42a2-istiod-ca-cert\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.486700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.486679    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-envoy\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.486943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.486925    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-podinfo\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.549853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.549823    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-token\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.550452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.550425    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6zh\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-kube-api-access-ts6zh\") pod \"e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") " pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.693021 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.692987    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"]
Oct 30 22:43:25.698976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.698955    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.744736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.744656    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"]
Oct 30 22:43:25.753400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.753373    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:25.753531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.753431    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:25.753915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.753887    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:25.787252 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.787221    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb96h\" (UniqueName: \"kubernetes.io/projected/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-kube-api-access-nb96h\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.787424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.787401    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.787579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.787552    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.787889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.787864    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:25.813109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.813087    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:25.850030 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.849994    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:43:25.888319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.888279    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.888591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.888361    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.888591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.888433    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb96h\" (UniqueName: \"kubernetes.io/projected/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-kube-api-access-nb96h\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.889420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.889387    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.896988 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.895166    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:25.959878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:25.959845    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb96h\" (UniqueName: \"kubernetes.io/projected/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-kube-api-access-nb96h\") pod \"maas-api-revoke-sub-d24c0f65-smbgn\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:26.021654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:26.021565    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:26.248142 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:43:26.248104    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc163a492_0c0a_4664_805b_5798e4cd42a2.slice/crio-1161a14f9d1682b86ddc32f4dcf148c22fd51d011fd44c4caebee4b253b4c413 WatchSource:0}: Error finding container 1161a14f9d1682b86ddc32f4dcf148c22fd51d011fd44c4caebee4b253b4c413: Status 404 returned error can't find the container with id 1161a14f9d1682b86ddc32f4dcf148c22fd51d011fd44c4caebee4b253b4c413
Oct 30 22:43:26.251237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:26.251195    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:43:26.251349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:26.251278    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:43:26.251349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:26.251312    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:43:26.251883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:26.251819    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"]
Oct 30 22:43:26.406399 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:43:26.406366    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ad5a3f_fd20_46b1_8c43_eb9086cd40f3.slice/crio-45187f0bd1a7449ae4ba4bea0b4ba94e98680eac7db4e9c40e4a0eb9b05c3071 WatchSource:0}: Error finding container 45187f0bd1a7449ae4ba4bea0b4ba94e98680eac7db4e9c40e4a0eb9b05c3071: Status 404 returned error can't find the container with id 45187f0bd1a7449ae4ba4bea0b4ba94e98680eac7db4e9c40e4a0eb9b05c3071
Oct 30 22:43:26.410033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:26.409435    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"]
Oct 30 22:43:27.178699 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.178662    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn" event={"ID":"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3","Type":"ContainerStarted","Data":"c9320bca6ce792117a33919f98ff32832457fe991d0a531e680d1ef96b4993b3"}
Oct 30 22:43:27.179145 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.178716    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn" event={"ID":"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3","Type":"ContainerStarted","Data":"45187f0bd1a7449ae4ba4bea0b4ba94e98680eac7db4e9c40e4a0eb9b05c3071"}
Oct 30 22:43:27.180074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.180048    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" event={"ID":"c163a492-0c0a-4664-805b-5798e4cd42a2","Type":"ContainerStarted","Data":"a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2"}
Oct 30 22:43:27.180190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.180088    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" event={"ID":"c163a492-0c0a-4664-805b-5798e4cd42a2","Type":"ContainerStarted","Data":"1161a14f9d1682b86ddc32f4dcf148c22fd51d011fd44c4caebee4b253b4c413"}
Oct 30 22:43:27.302934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.302881    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn" podStartSLOduration=2.302867245 podStartE2EDuration="2.302867245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:27.302452647 +0000 UTC m=+3383.637029417" watchObservedRunningTime="2026-10-30 22:43:27.302867245 +0000 UTC m=+3383.637444013"
Oct 30 22:43:27.436419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.436302    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podStartSLOduration=2.436287293 podStartE2EDuration="2.436287293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:27.43511031 +0000 UTC m=+3383.769687078" watchObservedRunningTime="2026-10-30 22:43:27.436287293 +0000 UTC m=+3383.770864062"
Oct 30 22:43:27.599866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.599815    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:27.813846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.813764    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:27.819957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:27.819934    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:28.104731 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:43:28.103777    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:43:28.187437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:28.187408    5503 generic.go:356] "Generic (PLEG): container finished" podID="44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" containerID="c9320bca6ce792117a33919f98ff32832457fe991d0a531e680d1ef96b4993b3" exitCode=0
Oct 30 22:43:28.187930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:28.187494    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn" event={"ID":"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3","Type":"ContainerDied","Data":"c9320bca6ce792117a33919f98ff32832457fe991d0a531e680d1ef96b4993b3"}
Oct 30 22:43:28.187930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:28.187854    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:28.188746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:28.188729    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:43:28.910316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:28.910283    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:29.072009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.071974    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:29.319874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.319843    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:29.345509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.345484    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:29.441131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.440101    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-xks-ca\") pod \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") "
Oct 30 22:43:29.441131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.440210    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-openshift-ca\") pod \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") "
Oct 30 22:43:29.441131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.440277    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-kube-api-access-nb96h\" (UniqueName: \"kubernetes.io/projected/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-kube-api-access-nb96h\") pod \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\" (UID: \"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3\") "
Oct 30 22:43:29.442213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.442165    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-openshift-ca" pod "44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" (UID: "44ad5a3f-fd20-46b1-8c43-eb9086cd40f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:29.450889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.450805    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-kube-api-access-nb96h" pod "44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" (UID: "44ad5a3f-fd20-46b1-8c43-eb9086cd40f3"). InnerVolumeSpecName "kube-api-access-nb96h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:29.452691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.452612    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-xks-ca" pod "44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" (UID: "44ad5a3f-fd20-46b1-8c43-eb9086cd40f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:29.492657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.492599    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:29.541582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.541530    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:29.541582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.541571    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:29.541582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.541582    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb96h\" (UniqueName: \"kubernetes.io/projected/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3-kube-api-access-nb96h\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:29.573134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.573100    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"]
Oct 30 22:43:29.576495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.576464    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cbe3a9-d52p5"]
Oct 30 22:43:29.789023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:29.788948    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c477a1b8-b540-4d05-bc9f-f3c7110f9124" path="/var/lib/kubelet/pods/c477a1b8-b540-4d05-bc9f-f3c7110f9124/volumes"
Oct 30 22:43:30.023518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:30.023485    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:30.188811 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:30.188771    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:30.199946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:30.199918    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"
Oct 30 22:43:30.200101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:30.199953    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn" event={"ID":"44ad5a3f-fd20-46b1-8c43-eb9086cd40f3","Type":"ContainerDied","Data":"45187f0bd1a7449ae4ba4bea0b4ba94e98680eac7db4e9c40e4a0eb9b05c3071"}
Oct 30 22:43:30.200101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:30.200028    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45187f0bd1a7449ae4ba4bea0b4ba94e98680eac7db4e9c40e4a0eb9b05c3071"
Oct 30 22:43:31.097662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:31.097620    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:31.319756 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:31.319724    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:31.486610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:31.486569    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:33.050747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:33.050707    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"]
Oct 30 22:43:33.101445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:33.101406    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e03ebd7-x8q62"]
Oct 30 22:43:33.790311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:33.790276    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451b0d73-a4ae-41c9-b52a-a042d73233d9" path="/var/lib/kubelet/pods/451b0d73-a4ae-41c9-b52a-a042d73233d9/volumes"
Oct 30 22:43:33.876857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:33.876821    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:34.373556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:34.373526    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:34.526806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:34.526776    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:34.677269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:34.677233    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:36.092385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.092353    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"]
Oct 30 22:43:36.093251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.093229    5503 cpu_manager.go:414] "containerMap: removing container" podUID="44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" containerName="revoke-keys"
Oct 30 22:43:36.093251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.093250    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" containerName="revoke-keys"
Oct 30 22:43:36.093457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.093362    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" containerName="revoke-keys"
Oct 30 22:43:36.095465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.095449    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.202544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.202514    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"]
Oct 30 22:43:36.239124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239088    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqkzr\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-kube-api-access-lqkzr\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239213    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343cb96d-6e76-4dd1-92ad-8794336aa759-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239263    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239283    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239303    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239326    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239439    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239510    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.239603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.239542    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340558    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340597    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340661    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqkzr\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-kube-api-access-lqkzr\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340714    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343cb96d-6e76-4dd1-92ad-8794336aa759-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340748    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340767    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340797    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340821    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.340881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.340847    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.341297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.341177    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-credential-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.341297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.341218    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-certs\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.341297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.341271    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-socket\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.341485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.341388    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-data\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.341485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.341455    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343cb96d-6e76-4dd1-92ad-8794336aa759-istiod-ca-cert\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.343077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.343025    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-envoy\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.343451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.343431    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-podinfo\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.402820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.402789    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-token\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.402961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.402828    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqkzr\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-kube-api-access-lqkzr\") pod \"e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") " pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.407628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.407604    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:36.433508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.433484    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"]
Oct 30 22:43:36.436494 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.436469    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.490076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.490042    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"]
Oct 30 22:43:36.490240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.490155    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:36.490240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.490173    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:43:36.490240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.490235    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:43:36.490435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.490272    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:43:36.543960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.543928    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.544141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.543978    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44kk\" (UniqueName: \"kubernetes.io/projected/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-kube-api-access-q44kk\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.544215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.544180    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.645659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.645556    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.645851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.645699    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.645851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.645740    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44kk\" (UniqueName: \"kubernetes.io/projected/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-kube-api-access-q44kk\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.646403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.646377    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.648071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.648047    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.717878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.717810    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44kk\" (UniqueName: \"kubernetes.io/projected/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-kube-api-access-q44kk\") pod \"maas-api-revoke-sub-fef97a57-mqc6n\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.752073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.752043    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:36.830479 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:43:36.830453    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343cb96d_6e76_4dd1_92ad_8794336aa759.slice/crio-64970a52d5bb734e85c306bb3d7fa5f2c2841a7b344a77b6cdb3457f24573ae0 WatchSource:0}: Error finding container 64970a52d5bb734e85c306bb3d7fa5f2c2841a7b344a77b6cdb3457f24573ae0: Status 404 returned error can't find the container with id 64970a52d5bb734e85c306bb3d7fa5f2c2841a7b344a77b6cdb3457f24573ae0
Oct 30 22:43:36.832834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.832811    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"]
Oct 30 22:43:36.833570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.833537    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:43:36.833645 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.833603    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:43:36.833697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:36.833654    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:43:37.117498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.117460    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:37.123842 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:43:37.123811    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5093ee1c_dbca_4e6b_8093_8c5087a54ebf.slice/crio-9f8d1f7523293005c30dee57fb0f12bb4a38229cf4a985e65d2063e7ccd5e5d6 WatchSource:0}: Error finding container 9f8d1f7523293005c30dee57fb0f12bb4a38229cf4a985e65d2063e7ccd5e5d6: Status 404 returned error can't find the container with id 9f8d1f7523293005c30dee57fb0f12bb4a38229cf4a985e65d2063e7ccd5e5d6
Oct 30 22:43:37.125694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.125572    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"]
Oct 30 22:43:37.247602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.247563    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n" event={"ID":"5093ee1c-dbca-4e6b-8093-8c5087a54ebf","Type":"ContainerStarted","Data":"9f8d1f7523293005c30dee57fb0f12bb4a38229cf4a985e65d2063e7ccd5e5d6"}
Oct 30 22:43:37.249618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.249593    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" event={"ID":"343cb96d-6e76-4dd1-92ad-8794336aa759","Type":"ContainerStarted","Data":"30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012"}
Oct 30 22:43:37.249774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.249675    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" event={"ID":"343cb96d-6e76-4dd1-92ad-8794336aa759","Type":"ContainerStarted","Data":"64970a52d5bb734e85c306bb3d7fa5f2c2841a7b344a77b6cdb3457f24573ae0"}
Oct 30 22:43:37.289251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.289204    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:37.375750 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.375690    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podStartSLOduration=1.3756688719999999 podStartE2EDuration="1.375668872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:37.37373633 +0000 UTC m=+3393.708313100" watchObservedRunningTime="2026-10-30 22:43:37.375668872 +0000 UTC m=+3393.710245639"
Oct 30 22:43:37.420874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.420837    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:37.430027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.430002    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 22:43:37.430160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.430056    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 22:43:37.478869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:37.478832    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:38.129460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:38.127041    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:38.258492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:38.258450    5503 generic.go:356] "Generic (PLEG): container finished" podID="5093ee1c-dbca-4e6b-8093-8c5087a54ebf" containerID="36980df9f373bc27fb62aecdbceb00012b15ecfbfa36d3ca83161b269182a249" exitCode=0
Oct 30 22:43:38.258735 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:38.258526    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n" event={"ID":"5093ee1c-dbca-4e6b-8093-8c5087a54ebf","Type":"ContainerDied","Data":"36980df9f373bc27fb62aecdbceb00012b15ecfbfa36d3ca83161b269182a249"}
Oct 30 22:43:38.408529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:38.408499    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 22:43:38.408726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:38.408554    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 22:43:38.790710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:38.790238    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:39.106488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.106379    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:39.366493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.351776    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:39.408894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.408401    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 22:43:39.408894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.408489    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 22:43:39.552863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.552832    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:39.705245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.705216    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-kube-api-access-q44kk\" (UniqueName: \"kubernetes.io/projected/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-kube-api-access-q44kk\") pod \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") "
Oct 30 22:43:39.705440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.705273    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-openshift-ca\") pod \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") "
Oct 30 22:43:39.705440 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.705319    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-xks-ca\") pod \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\" (UID: \"5093ee1c-dbca-4e6b-8093-8c5087a54ebf\") "
Oct 30 22:43:39.705681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.705647    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-openshift-ca" pod "5093ee1c-dbca-4e6b-8093-8c5087a54ebf" (UID: "5093ee1c-dbca-4e6b-8093-8c5087a54ebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:39.705813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.705790    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:39.707497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.707472    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-xks-ca" pod "5093ee1c-dbca-4e6b-8093-8c5087a54ebf" (UID: "5093ee1c-dbca-4e6b-8093-8c5087a54ebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:39.707601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.707504    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-kube-api-access-q44kk" pod "5093ee1c-dbca-4e6b-8093-8c5087a54ebf" (UID: "5093ee1c-dbca-4e6b-8093-8c5087a54ebf"). InnerVolumeSpecName "kube-api-access-q44kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:39.806784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.806753    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44kk\" (UniqueName: \"kubernetes.io/projected/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-kube-api-access-q44kk\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:39.806784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:39.806782    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5093ee1c-dbca-4e6b-8093-8c5087a54ebf-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:40.293789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.293750    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n" event={"ID":"5093ee1c-dbca-4e6b-8093-8c5087a54ebf","Type":"ContainerDied","Data":"9f8d1f7523293005c30dee57fb0f12bb4a38229cf4a985e65d2063e7ccd5e5d6"}
Oct 30 22:43:40.294149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.294130    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8d1f7523293005c30dee57fb0f12bb4a38229cf4a985e65d2063e7ccd5e5d6"
Oct 30 22:43:40.294284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.294082    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"
Oct 30 22:43:40.408042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.408001    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 22:43:40.408557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.408062    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 22:43:40.490948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.490916    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:40.651937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.651895    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:40.808601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:40.808560    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:41.408982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:41.408948    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 22:43:41.409404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:41.409002    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 22:43:42.409014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:42.408974    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 22:43:42.409431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:42.409049    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 22:43:43.460574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:43.460519    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:43.660342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:43.660304    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:43.661108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:43.661080    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:43.661857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:43.661833    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:43:44.392492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:44.388305    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:44.599767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:44.599731    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:44.902694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:44.902660    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:45.754421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:45.754383    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:45.931672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:45.931612    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:49.308861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:49.308825    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:49.957926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:49.957883    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:50.139974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:50.139945    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:50.351283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:50.351136    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:51.208072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:51.208001    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:54.155701 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:54.155662    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:55.318614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.318577    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:55.503796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.503761    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:43:55.743851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.743821    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"]
Oct 30 22:43:55.744349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.744334    5503 cpu_manager.go:414] "containerMap: removing container" podUID="5093ee1c-dbca-4e6b-8093-8c5087a54ebf" containerName="revoke-keys"
Oct 30 22:43:55.744396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.744353    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5093ee1c-dbca-4e6b-8093-8c5087a54ebf" containerName="revoke-keys"
Oct 30 22:43:55.744483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.744473    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="5093ee1c-dbca-4e6b-8093-8c5087a54ebf" containerName="revoke-keys"
Oct 30 22:43:55.748101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.748083    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:55.802206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.802181    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1b5075\""
Oct 30 22:43:55.852884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.852849    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1b5075-dockercfg-b5ljq\""
Oct 30 22:43:55.856939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.856907    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"]
Oct 30 22:43:55.876516 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.876462    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/d9e53a02-7158-4bc0-91b3-177c5e587222-kube-api-access-whmzd\") pod \"payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:55.876748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.876545    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e53a02-7158-4bc0-91b3-177c5e587222-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:55.915958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.915925    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"]
Oct 30 22:43:55.945703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.939686    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"]
Oct 30 22:43:55.946687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.939840    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:55.977659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.977511    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e53a02-7158-4bc0-91b3-177c5e587222-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:55.978697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.978612    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e53a02-7158-4bc0-91b3-177c5e587222-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:55.978909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:55.978882    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/d9e53a02-7158-4bc0-91b3-177c5e587222-kube-api-access-whmzd\") pod \"payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:56.043662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.043576    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/d9e53a02-7158-4bc0-91b3-177c5e587222-kube-api-access-whmzd\") pod \"payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:56.063653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.063604    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:56.080127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.080092    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6541ed5e-f464-4091-93aa-b8270ff2e4dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:56.080405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.080386    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppsz\" (UniqueName: \"kubernetes.io/projected/6541ed5e-f464-4091-93aa-b8270ff2e4dc-kube-api-access-zppsz\") pod \"payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:56.181653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.181588    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6541ed5e-f464-4091-93aa-b8270ff2e4dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:56.181843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.181781    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppsz\" (UniqueName: \"kubernetes.io/projected/6541ed5e-f464-4091-93aa-b8270ff2e4dc-kube-api-access-zppsz\") pod \"payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:56.182370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.182342    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6541ed5e-f464-4091-93aa-b8270ff2e4dc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:56.246080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.246047    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppsz\" (UniqueName: \"kubernetes.io/projected/6541ed5e-f464-4091-93aa-b8270ff2e4dc-kube-api-access-zppsz\") pod \"payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:56.284510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.284481    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:56.438736 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:43:56.438704    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e53a02_7158_4bc0_91b3_177c5e587222.slice/crio-891a036eae936138f51a95744e8bcf0c818e3f511647147a4d17e3fcf53fd6f0 WatchSource:0}: Error finding container 891a036eae936138f51a95744e8bcf0c818e3f511647147a4d17e3fcf53fd6f0: Status 404 returned error can't find the container with id 891a036eae936138f51a95744e8bcf0c818e3f511647147a4d17e3fcf53fd6f0
Oct 30 22:43:56.442153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.442130    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"]
Oct 30 22:43:56.463239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.463209    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" event={"ID":"d9e53a02-7158-4bc0-91b3-177c5e587222","Type":"ContainerStarted","Data":"891a036eae936138f51a95744e8bcf0c818e3f511647147a4d17e3fcf53fd6f0"}
Oct 30 22:43:56.654805 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:43:56.654771    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6541ed5e_f464_4091_93aa_b8270ff2e4dc.slice/crio-0f48a01484892ea4981f7d87efda13dc3eeb5c02218b5c41f92ae055f5b65c44 WatchSource:0}: Error finding container 0f48a01484892ea4981f7d87efda13dc3eeb5c02218b5c41f92ae055f5b65c44: Status 404 returned error can't find the container with id 0f48a01484892ea4981f7d87efda13dc3eeb5c02218b5c41f92ae055f5b65c44
Oct 30 22:43:56.657338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:56.657310    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"]
Oct 30 22:43:57.471269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:57.471233    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" event={"ID":"6541ed5e-f464-4091-93aa-b8270ff2e4dc","Type":"ContainerStarted","Data":"3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227"}
Oct 30 22:43:57.471779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:57.471286    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" event={"ID":"6541ed5e-f464-4091-93aa-b8270ff2e4dc","Type":"ContainerStarted","Data":"0f48a01484892ea4981f7d87efda13dc3eeb5c02218b5c41f92ae055f5b65c44"}
Oct 30 22:43:57.471779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:57.471439    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:43:57.472972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:57.472947    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" event={"ID":"d9e53a02-7158-4bc0-91b3-177c5e587222","Type":"ContainerStarted","Data":"d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b"}
Oct 30 22:43:57.473090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:57.473038    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:43:57.593411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:57.593354    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" podStartSLOduration=2.593342459 podStartE2EDuration="2.593342459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:57.592840885 +0000 UTC m=+3413.927417654" watchObservedRunningTime="2026-10-30 22:43:57.593342459 +0000 UTC m=+3413.927919227"
Oct 30 22:43:57.725169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:57.722443    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" podStartSLOduration=2.7224236619999997 podStartE2EDuration="2.722423662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:57.722025859 +0000 UTC m=+3414.056602630" watchObservedRunningTime="2026-10-30 22:43:57.722423662 +0000 UTC m=+3414.057000432"
Oct 30 22:43:58.099163 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:43:58.099086    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:43:58.827279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:43:58.827249    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:03.382189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:03.382158    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:05.780914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:05.780882    5503 scope.go:122] "RemoveContainer" containerID="04af6fad708c97226087ce59a76af71cdfaed8c2b02eab9de6a471889754457b"
Oct 30 22:44:07.911989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:07.911956    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:08.482040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:08.482013    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:44:08.482232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:08.482082    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:44:12.437194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:12.437159    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:14.990085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:14.990053    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:44:17.076785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:17.076752    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:17.680827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:17.680790    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:17.858591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:17.858549    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:21.765881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:21.765846    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:22.487986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:22.487953    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:23.315125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:23.315075    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:23.528810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:23.528771    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:26.612661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:26.612611    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:27.504498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:27.504455    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:27.770341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:27.770252    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:28.058118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:28.058039    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:28.098956 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:44:28.098924    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:44:29.258893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:29.258857    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:30.474747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:30.474705    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:30.768731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:30.768661    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:30.971999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:30.971967    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:31.322687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.322654    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"]
Oct 30 22:44:31.345926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.345891    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.388129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.388099    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"]
Oct 30 22:44:31.403359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.403315    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwz5\" (UniqueName: \"kubernetes.io/projected/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-kube-api-access-6dwz5\") pod \"payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") " pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.415658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.413808    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-plugins-config-volume\") pod \"payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") " pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.415658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.410685    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-dcde03-dockercfg-gds6b\""
Oct 30 22:44:31.415658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.410546    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-dcde03\""
Oct 30 22:44:31.515329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.515290    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwz5\" (UniqueName: \"kubernetes.io/projected/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-kube-api-access-6dwz5\") pod \"payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") " pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.515794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.515380    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-plugins-config-volume\") pod \"payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") " pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.516140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.516115    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-plugins-config-volume\") pod \"payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") " pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.577966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.577896    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwz5\" (UniqueName: \"kubernetes.io/projected/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-kube-api-access-6dwz5\") pod \"payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") " pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.691091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.691059    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:31.838269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:31.838177    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:32.062309 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:44:32.062276    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fa386b_eb7c_48e7_b46e_5e417fe0ed6c.slice/crio-a6928f8478570f1f92c7f45e248f625b7af716f8068d2d78ea3ada3891bb73ae WatchSource:0}: Error finding container a6928f8478570f1f92c7f45e248f625b7af716f8068d2d78ea3ada3891bb73ae: Status 404 returned error can't find the container with id a6928f8478570f1f92c7f45e248f625b7af716f8068d2d78ea3ada3891bb73ae
Oct 30 22:44:32.065157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:32.065136    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"]
Oct 30 22:44:32.705841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:32.705805    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8" event={"ID":"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c","Type":"ContainerStarted","Data":"670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657"}
Oct 30 22:44:32.706265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:32.705865    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:32.706265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:32.705886    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8" event={"ID":"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c","Type":"ContainerStarted","Data":"a6928f8478570f1f92c7f45e248f625b7af716f8068d2d78ea3ada3891bb73ae"}
Oct 30 22:44:32.825190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:32.825131    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8" podStartSLOduration=1.825118624 podStartE2EDuration="1.825118624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:32.824101193 +0000 UTC m=+3449.158677963" watchObservedRunningTime="2026-10-30 22:44:32.825118624 +0000 UTC m=+3449.159695395"
Oct 30 22:44:36.387494 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:36.387459    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:38.290681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:38.290622    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:39.509942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:39.509902    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"]
Oct 30 22:44:39.510409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:39.510153    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" containerID="cri-o://9d28507300ace1bb24f93e8b0ac6a125ec258b4bcc4bc26e0b8942e1947f28f7" gracePeriod=30
Oct 30 22:44:40.959410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:40.959375    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:43.718290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:43.718260    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:44:44.793150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:44.793072    5503 generic.go:356] "Generic (PLEG): container finished" podID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerID="9d28507300ace1bb24f93e8b0ac6a125ec258b4bcc4bc26e0b8942e1947f28f7" exitCode=0
Oct 30 22:44:44.793530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:44.793141    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" event={"ID":"19dd129c-14f1-4dd1-984a-f08dc42f64fd","Type":"ContainerDied","Data":"9d28507300ace1bb24f93e8b0ac6a125ec258b4bcc4bc26e0b8942e1947f28f7"}
Oct 30 22:44:45.076863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.076780    5503 patch_prober.go:29] interesting pod/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:44:45.077030 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.076861    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 22:44:45.162230 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.162207    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:44:45.270861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.270825    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-podinfo\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271030 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.270902    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-token\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-token\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271030 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.270945    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-data\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271049    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-certs\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271144    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-credential-socket\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271191    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istiod-ca-cert\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271231    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-envoy\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271269    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-socket\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271290    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-data" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:45.271397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271362    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-kube-api-access-5zrkd\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-kube-api-access-5zrkd\") pod \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\" (UID: \"19dd129c-14f1-4dd1-984a-f08dc42f64fd\") "
Oct 30 22:44:45.271397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271335    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-certs" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:45.271569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271418    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-credential-socket" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:45.271569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271555    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-socket" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:45.273456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271603    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istiod-ca-cert" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:45.273456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271879    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.273456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271901    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.273456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271919    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.273456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271935    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.273456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.271949    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.273867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.273487    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-token" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:45.273867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.273778    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-podinfo" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:44:45.273867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.273790    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-envoy" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:45.273867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.273828    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-kube-api-access-5zrkd" pod "19dd129c-14f1-4dd1-984a-f08dc42f64fd" (UID: "19dd129c-14f1-4dd1-984a-f08dc42f64fd"). InnerVolumeSpecName "kube-api-access-5zrkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:45.373596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.373526    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.373596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.373554    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrkd\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-kube-api-access-5zrkd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.373596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.373565    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.373596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.373573    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd129c-14f1-4dd1-984a-f08dc42f64fd-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:44:45.478041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.478004    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:45.641138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.641059    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:45.801048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.801022    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"
Oct 30 22:44:45.801410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.801045    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs" event={"ID":"19dd129c-14f1-4dd1-984a-f08dc42f64fd","Type":"ContainerDied","Data":"a8b856c2277d3bce7bacfc4745f5c7b5e8ba0098d123abf7f614abe86d842fa8"}
Oct 30 22:44:45.801410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.801100    5503 scope.go:122] "RemoveContainer" containerID="9d28507300ace1bb24f93e8b0ac6a125ec258b4bcc4bc26e0b8942e1947f28f7"
Oct 30 22:44:45.827981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.827957    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:45.976658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.976605    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"]
Oct 30 22:44:45.989542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:45.989512    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e65019-openshift-default-7b9c7cd659-wgtrs"]
Oct 30 22:44:47.789987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:47.789953    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" path="/var/lib/kubelet/pods/19dd129c-14f1-4dd1-984a-f08dc42f64fd/volumes"
Oct 30 22:44:48.800890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:48.800855    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:50.253476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:50.253441    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:51.052596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:51.052562    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"]
Oct 30 22:44:51.103355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:51.103324    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb6c7e75-hjfcz"]
Oct 30 22:44:51.789174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:51.789140    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997d6fa9-8222-42eb-b4a1-ac9bf0290185" path="/var/lib/kubelet/pods/997d6fa9-8222-42eb-b4a1-ac9bf0290185/volumes"
Oct 30 22:44:54.817073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:54.817046    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:55.761526 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:55.761484    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.092709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:56.091576    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.350472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:56.350390    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:56.516967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:56.516932    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:44:58.098991 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:44:58.098957    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:44:59.397502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:44:59.397465    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:00.168146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.168109    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"]
Oct 30 22:45:00.168702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.168687    5503 cpu_manager.go:414] "containerMap: removing container" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy"
Oct 30 22:45:00.168755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.168704    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy"
Oct 30 22:45:00.168816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.168807    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dd129c-14f1-4dd1-984a-f08dc42f64fd" containerName="istio-proxy"
Oct 30 22:45:00.173591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.173567    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.177623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.177600    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"]
Oct 30 22:45:00.181343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.181326    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"]
Oct 30 22:45:00.181443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.181427    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.185309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.185291    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.219230 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.219205    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"]
Oct 30 22:45:00.223099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.223077    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"]
Oct 30 22:45:00.223236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.223214    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.227342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.227316    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.228622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.228595    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"]
Oct 30 22:45:00.233007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.232987    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"]
Oct 30 22:45:00.233102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.233072    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.236768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.236749    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"]
Oct 30 22:45:00.236873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.236861    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.241416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.241395    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"]
Oct 30 22:45:00.241523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.241483    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.246096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.246075    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"]
Oct 30 22:45:00.246201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.246183    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.250589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.250562    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.263835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.263814    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e8631-01a7-4c0d-9aa1-2fc504473f92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.263943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.263845    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwj9\" (UniqueName: \"kubernetes.io/projected/d79579bf-99bc-4260-a73e-061c181cb66a-kube-api-access-lnwj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.263943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.263879    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8mv\" (UniqueName: \"kubernetes.io/projected/d649e45e-f885-42d9-82e8-4b03d3ed144d-kube-api-access-tn8mv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.264029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.264013    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzv89\" (UniqueName: \"kubernetes.io/projected/a50e8631-01a7-4c0d-9aa1-2fc504473f92-kube-api-access-bzv89\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.264090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.264077    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d649e45e-f885-42d9-82e8-4b03d3ed144d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.264185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.264160    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79579bf-99bc-4260-a73e-061c181cb66a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.267414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.267390    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:45:00.267525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.267479    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:45:00.270214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.270191    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"]
Oct 30 22:45:00.274444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.274421    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.276852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.276834    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:45:00.280013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.279991    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"]
Oct 30 22:45:00.282778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.282763    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"]
Oct 30 22:45:00.285376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.285358    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"]
Oct 30 22:45:00.323219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.319255    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"]
Oct 30 22:45:00.323579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.323541    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"]
Oct 30 22:45:00.327205 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.327181    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"]
Oct 30 22:45:00.329946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.329922    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"]
Oct 30 22:45:00.332115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.332091    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"]
Oct 30 22:45:00.359682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.359630    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"]
Oct 30 22:45:00.365420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365395    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwj9\" (UniqueName: \"kubernetes.io/projected/d79579bf-99bc-4260-a73e-061c181cb66a-kube-api-access-lnwj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.365557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365430    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.365557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365459    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33919e28-59ee-4a39-884b-59dc335abf8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.365557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365483    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8mv\" (UniqueName: \"kubernetes.io/projected/d649e45e-f885-42d9-82e8-4b03d3ed144d-kube-api-access-tn8mv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.365771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365601    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/399b48c0-01d3-47b7-8238-4b1b7f650212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.365771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365664    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942a698d-acea-43ab-9f76-f4b3e920a8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-h7rxm\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.365771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365707    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67g57\" (UniqueName: \"kubernetes.io/projected/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-kube-api-access-67g57\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.365971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365813    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5wf\" (UniqueName: \"kubernetes.io/projected/942a698d-acea-43ab-9f76-f4b3e920a8f1-kube-api-access-hc5wf\") pod \"maas-api-key-cleanup-29890005-h7rxm\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.365971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365925    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzv89\" (UniqueName: \"kubernetes.io/projected/a50e8631-01a7-4c0d-9aa1-2fc504473f92-kube-api-access-bzv89\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.366105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.365966    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c245e74f-8e3a-4898-9591-6ee505db9c7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.366105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366075    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128f1827-884b-4a6e-bc52-ef41d84ddbc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.366235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366110    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcl8\" (UniqueName: \"kubernetes.io/projected/128f1827-884b-4a6e-bc52-ef41d84ddbc0-kube-api-access-tjcl8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.366235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366187    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d649e45e-f885-42d9-82e8-4b03d3ed144d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.366235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366209    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsc5\" (UniqueName: \"kubernetes.io/projected/33919e28-59ee-4a39-884b-59dc335abf8c-kube-api-access-qgsc5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.366235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366234    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79579bf-99bc-4260-a73e-061c181cb66a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.366445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366315    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66r2t\" (UniqueName: \"kubernetes.io/projected/399b48c0-01d3-47b7-8238-4b1b7f650212-kube-api-access-66r2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.366445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366365    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.366560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366524    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894tr\" (UniqueName: \"kubernetes.io/projected/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-kube-api-access-894tr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.366621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366558    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjtj8\" (UniqueName: \"kubernetes.io/projected/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-kube-api-access-vjtj8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.366621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366609    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.366770 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366737    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e8631-01a7-4c0d-9aa1-2fc504473f92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.366835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366768    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n8z\" (UniqueName: \"kubernetes.io/projected/c245e74f-8e3a-4898-9591-6ee505db9c7d-kube-api-access-v2n8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.366950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.366925    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79579bf-99bc-4260-a73e-061c181cb66a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.367086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.367002    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d649e45e-f885-42d9-82e8-4b03d3ed144d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.367546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.367521    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e8631-01a7-4c0d-9aa1-2fc504473f92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.376620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.376592    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"]
Oct 30 22:45:00.380562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.380537    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"]
Oct 30 22:45:00.430284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.430198    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzv89\" (UniqueName: \"kubernetes.io/projected/a50e8631-01a7-4c0d-9aa1-2fc504473f92-kube-api-access-bzv89\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.430284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.430253    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8mv\" (UniqueName: \"kubernetes.io/projected/d649e45e-f885-42d9-82e8-4b03d3ed144d-kube-api-access-tn8mv\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.430840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.430532    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwj9\" (UniqueName: \"kubernetes.io/projected/d79579bf-99bc-4260-a73e-061c181cb66a-kube-api-access-lnwj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.468278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.468237    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.468501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.468304    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33919e28-59ee-4a39-884b-59dc335abf8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.468501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.468373    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/399b48c0-01d3-47b7-8238-4b1b7f650212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.468501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.468408    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942a698d-acea-43ab-9f76-f4b3e920a8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-h7rxm\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.468501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.468442    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67g57\" (UniqueName: \"kubernetes.io/projected/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-kube-api-access-67g57\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.468501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.468471    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5wf\" (UniqueName: \"kubernetes.io/projected/942a698d-acea-43ab-9f76-f4b3e920a8f1-kube-api-access-hc5wf\") pod \"maas-api-key-cleanup-29890005-h7rxm\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.469143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.469115    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.469926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.469896    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33919e28-59ee-4a39-884b-59dc335abf8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.469926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.469421    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942a698d-acea-43ab-9f76-f4b3e920a8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-h7rxm\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.470186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470149    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c245e74f-8e3a-4898-9591-6ee505db9c7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.470280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470200    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128f1827-884b-4a6e-bc52-ef41d84ddbc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.470280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470256    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcl8\" (UniqueName: \"kubernetes.io/projected/128f1827-884b-4a6e-bc52-ef41d84ddbc0-kube-api-access-tjcl8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.470445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470318    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsc5\" (UniqueName: \"kubernetes.io/projected/33919e28-59ee-4a39-884b-59dc335abf8c-kube-api-access-qgsc5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.470445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470356    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66r2t\" (UniqueName: \"kubernetes.io/projected/399b48c0-01d3-47b7-8238-4b1b7f650212-kube-api-access-66r2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.470445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470389    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.470698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470440    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894tr\" (UniqueName: \"kubernetes.io/projected/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-kube-api-access-894tr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.470698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470476    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjtj8\" (UniqueName: \"kubernetes.io/projected/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-kube-api-access-vjtj8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.470698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470532    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.470698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470571    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n8z\" (UniqueName: \"kubernetes.io/projected/c245e74f-8e3a-4898-9591-6ee505db9c7d-kube-api-access-v2n8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.470698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470612    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/399b48c0-01d3-47b7-8238-4b1b7f650212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.471119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.470915    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c245e74f-8e3a-4898-9591-6ee505db9c7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.471228 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.471166    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128f1827-884b-4a6e-bc52-ef41d84ddbc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.471381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.471349    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.471617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.471591    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.488405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.488380    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:00.497265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.497241    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:00.504367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.504342    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:00.535151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.534186    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjtj8\" (UniqueName: \"kubernetes.io/projected/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-kube-api-access-vjtj8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.535151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.534786    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n8z\" (UniqueName: \"kubernetes.io/projected/c245e74f-8e3a-4898-9591-6ee505db9c7d-kube-api-access-v2n8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.536542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.536481    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsc5\" (UniqueName: \"kubernetes.io/projected/33919e28-59ee-4a39-884b-59dc335abf8c-kube-api-access-qgsc5\") pod \"maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.537445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.537410    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894tr\" (UniqueName: \"kubernetes.io/projected/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-kube-api-access-894tr\") pod \"maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.537921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.537866    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcl8\" (UniqueName: \"kubernetes.io/projected/128f1827-884b-4a6e-bc52-ef41d84ddbc0-kube-api-access-tjcl8\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.539741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.538253    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67g57\" (UniqueName: \"kubernetes.io/projected/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-kube-api-access-67g57\") pod \"maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.539741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.538838    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66r2t\" (UniqueName: \"kubernetes.io/projected/399b48c0-01d3-47b7-8238-4b1b7f650212-kube-api-access-66r2t\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.543315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.543263    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5wf\" (UniqueName: \"kubernetes.io/projected/942a698d-acea-43ab-9f76-f4b3e920a8f1-kube-api-access-hc5wf\") pod \"maas-api-key-cleanup-29890005-h7rxm\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.558573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.558357    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:00.563740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.562626    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:00.573412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.567835    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:00.578157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.577841    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:00.590264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.585766    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:00.591959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.591773    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:00.600247 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.599700    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:00.837891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.837866    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:00.888086 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:00.888060    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50e8631_01a7_4c0d_9aa1_2fc504473f92.slice/crio-49b8ac58293541871f439781e16049da210c38cc33fd4d448a8e7165e53d91b5 WatchSource:0}: Error finding container 49b8ac58293541871f439781e16049da210c38cc33fd4d448a8e7165e53d91b5: Status 404 returned error can't find the container with id 49b8ac58293541871f439781e16049da210c38cc33fd4d448a8e7165e53d91b5
Oct 30 22:45:00.891679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.891655    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"]
Oct 30 22:45:00.955503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:00.955431    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" event={"ID":"a50e8631-01a7-4c0d-9aa1-2fc504473f92","Type":"ContainerStarted","Data":"49b8ac58293541871f439781e16049da210c38cc33fd4d448a8e7165e53d91b5"}
Oct 30 22:45:01.091367 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.091335    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79579bf_99bc_4260_a73e_061c181cb66a.slice/crio-5f0dd7fcaab0cae2ae97c764a1d7db8f13cf931d485449417c0da68f65a47be8 WatchSource:0}: Error finding container 5f0dd7fcaab0cae2ae97c764a1d7db8f13cf931d485449417c0da68f65a47be8: Status 404 returned error can't find the container with id 5f0dd7fcaab0cae2ae97c764a1d7db8f13cf931d485449417c0da68f65a47be8
Oct 30 22:45:01.094114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.094080    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"]
Oct 30 22:45:01.139987 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.139944    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd649e45e_f885_42d9_82e8_4b03d3ed144d.slice/crio-2b693605c01db9a2cb707e436cd31266462816c34975053b027be1ae4bcdf7eb WatchSource:0}: Error finding container 2b693605c01db9a2cb707e436cd31266462816c34975053b027be1ae4bcdf7eb: Status 404 returned error can't find the container with id 2b693605c01db9a2cb707e436cd31266462816c34975053b027be1ae4bcdf7eb
Oct 30 22:45:01.142854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.141976    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"]
Oct 30 22:45:01.146072 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.146045    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128f1827_884b_4a6e_bc52_ef41d84ddbc0.slice/crio-76611fb28ed73621af7db5ad00f1caa7e5fa2d156e8f667a19a5d8e743472cf9 WatchSource:0}: Error finding container 76611fb28ed73621af7db5ad00f1caa7e5fa2d156e8f667a19a5d8e743472cf9: Status 404 returned error can't find the container with id 76611fb28ed73621af7db5ad00f1caa7e5fa2d156e8f667a19a5d8e743472cf9
Oct 30 22:45:01.146534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.146507    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"]
Oct 30 22:45:01.149730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.149680    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"]
Oct 30 22:45:01.243745 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.243538    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc8ac2f_dd6e_40c8_8f1d_d8e4207000b7.slice/crio-4d612c03cc22b8c44c9334e9397c50f238508f8f14a63629b6b31cfba56e0161 WatchSource:0}: Error finding container 4d612c03cc22b8c44c9334e9397c50f238508f8f14a63629b6b31cfba56e0161: Status 404 returned error can't find the container with id 4d612c03cc22b8c44c9334e9397c50f238508f8f14a63629b6b31cfba56e0161
Oct 30 22:45:01.246047 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.246027    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc345c1_a266_4e1d_a688_ec82e72cf4e5.slice/crio-4a0c6b6ea517c735818ef215939de99ae296ebaeebf0631fcf615a7c332a4933 WatchSource:0}: Error finding container 4a0c6b6ea517c735818ef215939de99ae296ebaeebf0631fcf615a7c332a4933: Status 404 returned error can't find the container with id 4a0c6b6ea517c735818ef215939de99ae296ebaeebf0631fcf615a7c332a4933
Oct 30 22:45:01.246289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.246264    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"]
Oct 30 22:45:01.249498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.249479    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"]
Oct 30 22:45:01.289822 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.289798    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33919e28_59ee_4a39_884b_59dc335abf8c.slice/crio-2a14e7e1bb7d490293db113222b13aa3fe4b19f2caa26f8fe68fd607d723c6b9 WatchSource:0}: Error finding container 2a14e7e1bb7d490293db113222b13aa3fe4b19f2caa26f8fe68fd607d723c6b9: Status 404 returned error can't find the container with id 2a14e7e1bb7d490293db113222b13aa3fe4b19f2caa26f8fe68fd607d723c6b9
Oct 30 22:45:01.294421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.293936    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"]
Oct 30 22:45:01.298759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.298739    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"]
Oct 30 22:45:01.342378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.342254    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"]
Oct 30 22:45:01.389008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.388986    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"]
Oct 30 22:45:01.419390 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.419357    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942a698d_acea_43ab_9f76_f4b3e920a8f1.slice/crio-a483b06ef471f50e4f1f9c5eda4d16afa381fa0d50f9b3b90187ff54dfe152cd WatchSource:0}: Error finding container a483b06ef471f50e4f1f9c5eda4d16afa381fa0d50f9b3b90187ff54dfe152cd: Status 404 returned error can't find the container with id a483b06ef471f50e4f1f9c5eda4d16afa381fa0d50f9b3b90187ff54dfe152cd
Oct 30 22:45:01.420847 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:01.420822    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc245e74f_8e3a_4898_9591_6ee505db9c7d.slice/crio-c085908bcb258488ff8c528e64086ac61c6cccc32c2b45f6d30a63eddb7adc0b WatchSource:0}: Error finding container c085908bcb258488ff8c528e64086ac61c6cccc32c2b45f6d30a63eddb7adc0b: Status 404 returned error can't find the container with id c085908bcb258488ff8c528e64086ac61c6cccc32c2b45f6d30a63eddb7adc0b
Oct 30 22:45:01.541843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.541813    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:01.963573 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.963536    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" event={"ID":"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6","Type":"ContainerStarted","Data":"a4bc094a9520f93550870a67c7f73dc22f545921955d8bbdd027907c3da434b9"}
Oct 30 22:45:01.963789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.963601    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" event={"ID":"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6","Type":"ContainerStarted","Data":"60b30e8d77b80d99efbf2130e8d70d7e36a4826cc281097ff73f24220657e99f"}
Oct 30 22:45:01.965203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.965175    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerStarted","Data":"e5959406a3ae1c4d33524b0794e88788573a346e9bffdacccf2e8c1a8a8ddb8a"}
Oct 30 22:45:01.965318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.965227    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerStarted","Data":"4d612c03cc22b8c44c9334e9397c50f238508f8f14a63629b6b31cfba56e0161"}
Oct 30 22:45:01.966395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.966373    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" event={"ID":"399b48c0-01d3-47b7-8238-4b1b7f650212","Type":"ContainerStarted","Data":"92988ca28cfa71887e4e194d15289d3ed28cde7836df12d21b0cd8162e040eeb"}
Oct 30 22:45:01.967533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.967512    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" event={"ID":"942a698d-acea-43ab-9f76-f4b3e920a8f1","Type":"ContainerStarted","Data":"a483b06ef471f50e4f1f9c5eda4d16afa381fa0d50f9b3b90187ff54dfe152cd"}
Oct 30 22:45:01.968998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.968970    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" event={"ID":"d649e45e-f885-42d9-82e8-4b03d3ed144d","Type":"ContainerStarted","Data":"41bcb027998e226830fc4af1382d6e42c35f338ad8fa73fe195702c94cd166cc"}
Oct 30 22:45:01.969088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.969015    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" event={"ID":"d649e45e-f885-42d9-82e8-4b03d3ed144d","Type":"ContainerStarted","Data":"2b693605c01db9a2cb707e436cd31266462816c34975053b027be1ae4bcdf7eb"}
Oct 30 22:45:01.970499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.970478    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerStarted","Data":"251c4798cdededb95e080c2a5c59f17274aafdb200d10cb2d0f76248227d12ae"}
Oct 30 22:45:01.970595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.970517    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerStarted","Data":"5f0dd7fcaab0cae2ae97c764a1d7db8f13cf931d485449417c0da68f65a47be8"}
Oct 30 22:45:01.971970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.971942    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" event={"ID":"33919e28-59ee-4a39-884b-59dc335abf8c","Type":"ContainerStarted","Data":"b3303235443a6319b3ccde9bdfcf641bf6f257cef75bb813e1520327ec620e1c"}
Oct 30 22:45:01.972099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.971980    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" event={"ID":"33919e28-59ee-4a39-884b-59dc335abf8c","Type":"ContainerStarted","Data":"2a14e7e1bb7d490293db113222b13aa3fe4b19f2caa26f8fe68fd607d723c6b9"}
Oct 30 22:45:01.973442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.973422    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" event={"ID":"0fc345c1-a266-4e1d-a688-ec82e72cf4e5","Type":"ContainerStarted","Data":"019fe0a451ce03fad5c4499d2525b8b88ef30d814e0e9379d6b2c65b0716652b"}
Oct 30 22:45:01.973525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.973461    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" event={"ID":"0fc345c1-a266-4e1d-a688-ec82e72cf4e5","Type":"ContainerStarted","Data":"4a0c6b6ea517c735818ef215939de99ae296ebaeebf0631fcf615a7c332a4933"}
Oct 30 22:45:01.974578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.974559    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" event={"ID":"c245e74f-8e3a-4898-9591-6ee505db9c7d","Type":"ContainerStarted","Data":"c085908bcb258488ff8c528e64086ac61c6cccc32c2b45f6d30a63eddb7adc0b"}
Oct 30 22:45:01.976005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.975984    5503 generic.go:356] "Generic (PLEG): container finished" podID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerID="efa34424c20a473d264990e2ebf5fc7dda9e0961bfc2cd85eb6ab4d5a8093ef2" exitCode=60
Oct 30 22:45:01.976103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.976044    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" event={"ID":"a50e8631-01a7-4c0d-9aa1-2fc504473f92","Type":"ContainerDied","Data":"efa34424c20a473d264990e2ebf5fc7dda9e0961bfc2cd85eb6ab4d5a8093ef2"}
Oct 30 22:45:01.976229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.976215    5503 scope.go:122] "RemoveContainer" containerID="efa34424c20a473d264990e2ebf5fc7dda9e0961bfc2cd85eb6ab4d5a8093ef2"
Oct 30 22:45:01.977819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.977624    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" event={"ID":"128f1827-884b-4a6e-bc52-ef41d84ddbc0","Type":"ContainerStarted","Data":"4ca9543c7a6c372d585b619be49bb6d42c9da74c4d6a771dda0138eba7385f52"}
Oct 30 22:45:01.977819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:01.977693    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" event={"ID":"128f1827-884b-4a6e-bc52-ef41d84ddbc0","Type":"ContainerStarted","Data":"76611fb28ed73621af7db5ad00f1caa7e5fa2d156e8f667a19a5d8e743472cf9"}
Oct 30 22:45:02.053361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.053297    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"]
Oct 30 22:45:02.056627 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.056605    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efca9136-ngtz6"]
Oct 30 22:45:02.084629 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.084576    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" podStartSLOduration=2.084563253 podStartE2EDuration="2.084563253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.08202286 +0000 UTC m=+3478.416599628" watchObservedRunningTime="2026-10-30 22:45:02.084563253 +0000 UTC m=+3478.419140022"
Oct 30 22:45:02.320823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.320766    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" podStartSLOduration=2.320749315 podStartE2EDuration="2.320749315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.3195861 +0000 UTC m=+3478.654162872" watchObservedRunningTime="2026-10-30 22:45:02.320749315 +0000 UTC m=+3478.655326085"
Oct 30 22:45:02.442959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.442914    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" podStartSLOduration=2.442900226 podStartE2EDuration="2.442900226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.442131334 +0000 UTC m=+3478.776708104" watchObservedRunningTime="2026-10-30 22:45:02.442900226 +0000 UTC m=+3478.777476994"
Oct 30 22:45:02.562677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.562610    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" podStartSLOduration=2.5625974339999997 podStartE2EDuration="2.562597434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.560129824 +0000 UTC m=+3478.894706594" watchObservedRunningTime="2026-10-30 22:45:02.562597434 +0000 UTC m=+3478.897174203"
Oct 30 22:45:02.679200 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.679150    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" podStartSLOduration=2.679136115 podStartE2EDuration="2.679136115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.677747067 +0000 UTC m=+3479.012323835" watchObservedRunningTime="2026-10-30 22:45:02.679136115 +0000 UTC m=+3479.013712884"
Oct 30 22:45:02.798981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.798923    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" podStartSLOduration=2.798906777 podStartE2EDuration="2.798906777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.796092557 +0000 UTC m=+3479.130669327" watchObservedRunningTime="2026-10-30 22:45:02.798906777 +0000 UTC m=+3479.133483589"
Oct 30 22:45:02.914233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.914180    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" podStartSLOduration=2.914166544 podStartE2EDuration="2.914166544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.913292278 +0000 UTC m=+3479.247869048" watchObservedRunningTime="2026-10-30 22:45:02.914166544 +0000 UTC m=+3479.248743313"
Oct 30 22:45:02.986964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.986930    5503 generic.go:356] "Generic (PLEG): container finished" podID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerID="a4bc094a9520f93550870a67c7f73dc22f545921955d8bbdd027907c3da434b9" exitCode=60
Oct 30 22:45:02.987114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.987028    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" event={"ID":"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6","Type":"ContainerDied","Data":"a4bc094a9520f93550870a67c7f73dc22f545921955d8bbdd027907c3da434b9"}
Oct 30 22:45:02.987312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.987291    5503 scope.go:122] "RemoveContainer" containerID="a4bc094a9520f93550870a67c7f73dc22f545921955d8bbdd027907c3da434b9"
Oct 30 22:45:02.988685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.988662    5503 generic.go:356] "Generic (PLEG): container finished" podID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerID="e5959406a3ae1c4d33524b0794e88788573a346e9bffdacccf2e8c1a8a8ddb8a" exitCode=60
Oct 30 22:45:02.988783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.988761    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerDied","Data":"e5959406a3ae1c4d33524b0794e88788573a346e9bffdacccf2e8c1a8a8ddb8a"}
Oct 30 22:45:02.989038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.988981    5503 scope.go:122] "RemoveContainer" containerID="e5959406a3ae1c4d33524b0794e88788573a346e9bffdacccf2e8c1a8a8ddb8a"
Oct 30 22:45:02.990832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.990785    5503 generic.go:356] "Generic (PLEG): container finished" podID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerID="aac98e3a8b8c93a8fbb667def227e1010434b3d6fdfee7ef6cf5ce77b94a036e" exitCode=60
Oct 30 22:45:02.990904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.990854    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" event={"ID":"399b48c0-01d3-47b7-8238-4b1b7f650212","Type":"ContainerDied","Data":"aac98e3a8b8c93a8fbb667def227e1010434b3d6fdfee7ef6cf5ce77b94a036e"}
Oct 30 22:45:02.991094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.991078    5503 scope.go:122] "RemoveContainer" containerID="aac98e3a8b8c93a8fbb667def227e1010434b3d6fdfee7ef6cf5ce77b94a036e"
Oct 30 22:45:02.992571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.992554    5503 generic.go:356] "Generic (PLEG): container finished" podID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerID="439f2162207bd15593a54b2e679335eec17b4ac07b9a671fe682b9af6999c97f" exitCode=60
Oct 30 22:45:02.992667 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.992596    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" event={"ID":"942a698d-acea-43ab-9f76-f4b3e920a8f1","Type":"ContainerDied","Data":"439f2162207bd15593a54b2e679335eec17b4ac07b9a671fe682b9af6999c97f"}
Oct 30 22:45:02.992840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.992828    5503 scope.go:122] "RemoveContainer" containerID="439f2162207bd15593a54b2e679335eec17b4ac07b9a671fe682b9af6999c97f"
Oct 30 22:45:02.994171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.994150    5503 generic.go:356] "Generic (PLEG): container finished" podID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerID="41bcb027998e226830fc4af1382d6e42c35f338ad8fa73fe195702c94cd166cc" exitCode=60
Oct 30 22:45:02.994260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.994224    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" event={"ID":"d649e45e-f885-42d9-82e8-4b03d3ed144d","Type":"ContainerDied","Data":"41bcb027998e226830fc4af1382d6e42c35f338ad8fa73fe195702c94cd166cc"}
Oct 30 22:45:02.994411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.994394    5503 scope.go:122] "RemoveContainer" containerID="41bcb027998e226830fc4af1382d6e42c35f338ad8fa73fe195702c94cd166cc"
Oct 30 22:45:02.995800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.995783    5503 generic.go:356] "Generic (PLEG): container finished" podID="d79579bf-99bc-4260-a73e-061c181cb66a" containerID="251c4798cdededb95e080c2a5c59f17274aafdb200d10cb2d0f76248227d12ae" exitCode=60
Oct 30 22:45:02.995896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.995848    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerDied","Data":"251c4798cdededb95e080c2a5c59f17274aafdb200d10cb2d0f76248227d12ae"}
Oct 30 22:45:02.996019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.996006    5503 scope.go:122] "RemoveContainer" containerID="251c4798cdededb95e080c2a5c59f17274aafdb200d10cb2d0f76248227d12ae"
Oct 30 22:45:02.997445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.997424    5503 generic.go:356] "Generic (PLEG): container finished" podID="33919e28-59ee-4a39-884b-59dc335abf8c" containerID="b3303235443a6319b3ccde9bdfcf641bf6f257cef75bb813e1520327ec620e1c" exitCode=60
Oct 30 22:45:02.997535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.997498    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" event={"ID":"33919e28-59ee-4a39-884b-59dc335abf8c","Type":"ContainerDied","Data":"b3303235443a6319b3ccde9bdfcf641bf6f257cef75bb813e1520327ec620e1c"}
Oct 30 22:45:02.997659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.997624    5503 scope.go:122] "RemoveContainer" containerID="b3303235443a6319b3ccde9bdfcf641bf6f257cef75bb813e1520327ec620e1c"
Oct 30 22:45:02.999387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.999368    5503 generic.go:356] "Generic (PLEG): container finished" podID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerID="019fe0a451ce03fad5c4499d2525b8b88ef30d814e0e9379d6b2c65b0716652b" exitCode=60
Oct 30 22:45:02.999472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.999455    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" event={"ID":"0fc345c1-a266-4e1d-a688-ec82e72cf4e5","Type":"ContainerDied","Data":"019fe0a451ce03fad5c4499d2525b8b88ef30d814e0e9379d6b2c65b0716652b"}
Oct 30 22:45:02.999686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:02.999630    5503 scope.go:122] "RemoveContainer" containerID="019fe0a451ce03fad5c4499d2525b8b88ef30d814e0e9379d6b2c65b0716652b"
Oct 30 22:45:03.001002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.000971    5503 generic.go:356] "Generic (PLEG): container finished" podID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerID="aadcc12e0b4f213a880b27fb8d333697502f4aa422dcae90fde8413de4e59220" exitCode=60
Oct 30 22:45:03.001089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.001042    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" event={"ID":"c245e74f-8e3a-4898-9591-6ee505db9c7d","Type":"ContainerDied","Data":"aadcc12e0b4f213a880b27fb8d333697502f4aa422dcae90fde8413de4e59220"}
Oct 30 22:45:03.001224 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.001208    5503 scope.go:122] "RemoveContainer" containerID="aadcc12e0b4f213a880b27fb8d333697502f4aa422dcae90fde8413de4e59220"
Oct 30 22:45:03.003195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.003174    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" event={"ID":"a50e8631-01a7-4c0d-9aa1-2fc504473f92","Type":"ContainerStarted","Data":"73f93973749b2b9b98421e9230543d5c9c01124f9024e562bbfd429b0b2f6168"}
Oct 30 22:45:03.004850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.004828    5503 generic.go:356] "Generic (PLEG): container finished" podID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerID="4ca9543c7a6c372d585b619be49bb6d42c9da74c4d6a771dda0138eba7385f52" exitCode=60
Oct 30 22:45:03.004939 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.004881    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" event={"ID":"128f1827-884b-4a6e-bc52-ef41d84ddbc0","Type":"ContainerDied","Data":"4ca9543c7a6c372d585b619be49bb6d42c9da74c4d6a771dda0138eba7385f52"}
Oct 30 22:45:03.005077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.005062    5503 scope.go:122] "RemoveContainer" containerID="4ca9543c7a6c372d585b619be49bb6d42c9da74c4d6a771dda0138eba7385f52"
Oct 30 22:45:03.791104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:03.791031    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6be266-a9f8-4938-8b03-1c258ebd83ce" path="/var/lib/kubelet/pods/6b6be266-a9f8-4938-8b03-1c258ebd83ce/volumes"
Oct 30 22:45:04.013901 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.013862    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" event={"ID":"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6","Type":"ContainerStarted","Data":"f3db9d284c80b8aa02357ce97457b16ff522d38ec1fa0d8b2a5d1049b0783539"}
Oct 30 22:45:04.022564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.022338    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerStarted","Data":"739589ea4525d2cdbb497cea411b98e1b1d0d1a068dd334576362b328c09ef09"}
Oct 30 22:45:04.024499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.024470    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" event={"ID":"399b48c0-01d3-47b7-8238-4b1b7f650212","Type":"ContainerStarted","Data":"5764fe52f8895a521e61169ed9004b6340901b6a571e3da88fe2301e248887fc"}
Oct 30 22:45:04.026400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.026378    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" event={"ID":"942a698d-acea-43ab-9f76-f4b3e920a8f1","Type":"ContainerStarted","Data":"5850fb343ebc28994b4901dbded9a5e0146f85fcbd377cbb4c19ed6e6f4a4068"}
Oct 30 22:45:04.028415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.028395    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" event={"ID":"d649e45e-f885-42d9-82e8-4b03d3ed144d","Type":"ContainerStarted","Data":"4b895377c34c8c387b12cc53d71b46830c1a372a59eab512dd568d8e7e16dfb0"}
Oct 30 22:45:04.030250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.030226    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerStarted","Data":"83d236eb81ace936cbf12d71d1b543ce4d2f3668cea10ca442e5a538fb777f40"}
Oct 30 22:45:04.032060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.032040    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" event={"ID":"33919e28-59ee-4a39-884b-59dc335abf8c","Type":"ContainerStarted","Data":"bcc4b2b13ef9ce068dc1d609fb8076774b576a8206693f2d31272ae8642241d0"}
Oct 30 22:45:04.034018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.033892    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" event={"ID":"0fc345c1-a266-4e1d-a688-ec82e72cf4e5","Type":"ContainerStarted","Data":"83e20e8b19181319f5669e9f85642a4929208c3533ef37a33147bcb840049e65"}
Oct 30 22:45:04.036107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.036086    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" event={"ID":"c245e74f-8e3a-4898-9591-6ee505db9c7d","Type":"ContainerStarted","Data":"3dce0ab346e6af3210c1f95c8f6ec6d04a9b29ffad93f05135d054965a69dbb6"}
Oct 30 22:45:04.037976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.037955    5503 generic.go:356] "Generic (PLEG): container finished" podID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerID="73f93973749b2b9b98421e9230543d5c9c01124f9024e562bbfd429b0b2f6168" exitCode=60
Oct 30 22:45:04.038068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.038031    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" event={"ID":"a50e8631-01a7-4c0d-9aa1-2fc504473f92","Type":"ContainerDied","Data":"73f93973749b2b9b98421e9230543d5c9c01124f9024e562bbfd429b0b2f6168"}
Oct 30 22:45:04.038134 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.038084    5503 scope.go:122] "RemoveContainer" containerID="efa34424c20a473d264990e2ebf5fc7dda9e0961bfc2cd85eb6ab4d5a8093ef2"
Oct 30 22:45:04.038207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.038151    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:04.038207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.038193    5503 scope.go:122] "RemoveContainer" containerID="73f93973749b2b9b98421e9230543d5c9c01124f9024e562bbfd429b0b2f6168"
Oct 30 22:45:04.038423 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:04.038403    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s_odh-ai-gateway-infra(a50e8631-01a7-4c0d-9aa1-2fc504473f92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92"
Oct 30 22:45:04.039891 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.039874    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" event={"ID":"128f1827-884b-4a6e-bc52-ef41d84ddbc0","Type":"ContainerStarted","Data":"5e4434af2019e3c60fdd50d4266db6bc5fe6b73785c692f0ff5fc8085018c778"}
Oct 30 22:45:04.057083 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.057015    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" podStartSLOduration=4.057003148 podStartE2EDuration="4.057003148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.054929338 +0000 UTC m=+3480.389506107" watchObservedRunningTime="2026-10-30 22:45:04.057003148 +0000 UTC m=+3480.391579916"
Oct 30 22:45:04.644935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.644887    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" podStartSLOduration=4.644873269 podStartE2EDuration="4.644873269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.64396974 +0000 UTC m=+3480.978546509" watchObservedRunningTime="2026-10-30 22:45:04.644873269 +0000 UTC m=+3480.979450038"
Oct 30 22:45:04.880102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:04.879999    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" podStartSLOduration=4.879985323 podStartE2EDuration="4.879985323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.87731875 +0000 UTC m=+3481.211895513" watchObservedRunningTime="2026-10-30 22:45:04.879985323 +0000 UTC m=+3481.214562108"
Oct 30 22:45:05.049938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.049897    5503 generic.go:356] "Generic (PLEG): container finished" podID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerID="739589ea4525d2cdbb497cea411b98e1b1d0d1a068dd334576362b328c09ef09" exitCode=60
Oct 30 22:45:05.050138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.049942    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerDied","Data":"739589ea4525d2cdbb497cea411b98e1b1d0d1a068dd334576362b328c09ef09"}
Oct 30 22:45:05.050138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.050014    5503 scope.go:122] "RemoveContainer" containerID="e5959406a3ae1c4d33524b0794e88788573a346e9bffdacccf2e8c1a8a8ddb8a"
Oct 30 22:45:05.050314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.050298    5503 scope.go:122] "RemoveContainer" containerID="739589ea4525d2cdbb497cea411b98e1b1d0d1a068dd334576362b328c09ef09"
Oct 30 22:45:05.050620 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.050599    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h_odh-ai-gateway-infra(2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7"
Oct 30 22:45:05.052142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.052118    5503 generic.go:356] "Generic (PLEG): container finished" podID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerID="5764fe52f8895a521e61169ed9004b6340901b6a571e3da88fe2301e248887fc" exitCode=60
Oct 30 22:45:05.052241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.052190    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" event={"ID":"399b48c0-01d3-47b7-8238-4b1b7f650212","Type":"ContainerDied","Data":"5764fe52f8895a521e61169ed9004b6340901b6a571e3da88fe2301e248887fc"}
Oct 30 22:45:05.052449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.052431    5503 scope.go:122] "RemoveContainer" containerID="5764fe52f8895a521e61169ed9004b6340901b6a571e3da88fe2301e248887fc"
Oct 30 22:45:05.052784 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.052764    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d_odh-ai-gateway-infra(399b48c0-01d3-47b7-8238-4b1b7f650212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212"
Oct 30 22:45:05.054125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.054101    5503 generic.go:356] "Generic (PLEG): container finished" podID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerID="5850fb343ebc28994b4901dbded9a5e0146f85fcbd377cbb4c19ed6e6f4a4068" exitCode=60
Oct 30 22:45:05.054241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.054139    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" event={"ID":"942a698d-acea-43ab-9f76-f4b3e920a8f1","Type":"ContainerDied","Data":"5850fb343ebc28994b4901dbded9a5e0146f85fcbd377cbb4c19ed6e6f4a4068"}
Oct 30 22:45:05.054412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.054394    5503 scope.go:122] "RemoveContainer" containerID="5850fb343ebc28994b4901dbded9a5e0146f85fcbd377cbb4c19ed6e6f4a4068"
Oct 30 22:45:05.054648 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.054617    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-h7rxm_odh-ai-gateway-infra(942a698d-acea-43ab-9f76-f4b3e920a8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1"
Oct 30 22:45:05.056079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.056056    5503 generic.go:356] "Generic (PLEG): container finished" podID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerID="4b895377c34c8c387b12cc53d71b46830c1a372a59eab512dd568d8e7e16dfb0" exitCode=60
Oct 30 22:45:05.056175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.056079    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" event={"ID":"d649e45e-f885-42d9-82e8-4b03d3ed144d","Type":"ContainerDied","Data":"4b895377c34c8c387b12cc53d71b46830c1a372a59eab512dd568d8e7e16dfb0"}
Oct 30 22:45:05.056330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.056318    5503 scope.go:122] "RemoveContainer" containerID="4b895377c34c8c387b12cc53d71b46830c1a372a59eab512dd568d8e7e16dfb0"
Oct 30 22:45:05.056544 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.056523    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw_odh-ai-gateway-infra(d649e45e-f885-42d9-82e8-4b03d3ed144d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d"
Oct 30 22:45:05.058108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.058086    5503 generic.go:356] "Generic (PLEG): container finished" podID="d79579bf-99bc-4260-a73e-061c181cb66a" containerID="83d236eb81ace936cbf12d71d1b543ce4d2f3668cea10ca442e5a538fb777f40" exitCode=60
Oct 30 22:45:05.058226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.058171    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerDied","Data":"83d236eb81ace936cbf12d71d1b543ce4d2f3668cea10ca442e5a538fb777f40"}
Oct 30 22:45:05.058367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.058352    5503 scope.go:122] "RemoveContainer" containerID="83d236eb81ace936cbf12d71d1b543ce4d2f3668cea10ca442e5a538fb777f40"
Oct 30 22:45:05.058566 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.058541    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5_odh-ai-gateway-infra(d79579bf-99bc-4260-a73e-061c181cb66a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" podUID="d79579bf-99bc-4260-a73e-061c181cb66a"
Oct 30 22:45:05.060174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.060149    5503 generic.go:356] "Generic (PLEG): container finished" podID="33919e28-59ee-4a39-884b-59dc335abf8c" containerID="bcc4b2b13ef9ce068dc1d609fb8076774b576a8206693f2d31272ae8642241d0" exitCode=60
Oct 30 22:45:05.060264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.060236    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" event={"ID":"33919e28-59ee-4a39-884b-59dc335abf8c","Type":"ContainerDied","Data":"bcc4b2b13ef9ce068dc1d609fb8076774b576a8206693f2d31272ae8642241d0"}
Oct 30 22:45:05.060408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.060395    5503 scope.go:122] "RemoveContainer" containerID="bcc4b2b13ef9ce068dc1d609fb8076774b576a8206693f2d31272ae8642241d0"
Oct 30 22:45:05.060613 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.060594    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp_odh-ai-gateway-infra(33919e28-59ee-4a39-884b-59dc335abf8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" podUID="33919e28-59ee-4a39-884b-59dc335abf8c"
Oct 30 22:45:05.062062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.062042    5503 generic.go:356] "Generic (PLEG): container finished" podID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerID="83e20e8b19181319f5669e9f85642a4929208c3533ef37a33147bcb840049e65" exitCode=60
Oct 30 22:45:05.062158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.062125    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" event={"ID":"0fc345c1-a266-4e1d-a688-ec82e72cf4e5","Type":"ContainerDied","Data":"83e20e8b19181319f5669e9f85642a4929208c3533ef37a33147bcb840049e65"}
Oct 30 22:45:05.062311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.062298    5503 scope.go:122] "RemoveContainer" containerID="83e20e8b19181319f5669e9f85642a4929208c3533ef37a33147bcb840049e65"
Oct 30 22:45:05.062507 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.062487    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp_odh-ai-gateway-infra(0fc345c1-a266-4e1d-a688-ec82e72cf4e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5"
Oct 30 22:45:05.063986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.063967    5503 generic.go:356] "Generic (PLEG): container finished" podID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerID="3dce0ab346e6af3210c1f95c8f6ec6d04a9b29ffad93f05135d054965a69dbb6" exitCode=60
Oct 30 22:45:05.064092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.064048    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" event={"ID":"c245e74f-8e3a-4898-9591-6ee505db9c7d","Type":"ContainerDied","Data":"3dce0ab346e6af3210c1f95c8f6ec6d04a9b29ffad93f05135d054965a69dbb6"}
Oct 30 22:45:05.064217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.064202    5503 scope.go:122] "RemoveContainer" containerID="3dce0ab346e6af3210c1f95c8f6ec6d04a9b29ffad93f05135d054965a69dbb6"
Oct 30 22:45:05.064420 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.064399    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn_odh-ai-gateway-infra(c245e74f-8e3a-4898-9591-6ee505db9c7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d"
Oct 30 22:45:05.065622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.065606    5503 scope.go:122] "RemoveContainer" containerID="aac98e3a8b8c93a8fbb667def227e1010434b3d6fdfee7ef6cf5ce77b94a036e"
Oct 30 22:45:05.065917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.065898    5503 scope.go:122] "RemoveContainer" containerID="73f93973749b2b9b98421e9230543d5c9c01124f9024e562bbfd429b0b2f6168"
Oct 30 22:45:05.066116 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.066097    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s_odh-ai-gateway-infra(a50e8631-01a7-4c0d-9aa1-2fc504473f92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92"
Oct 30 22:45:05.067565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.067547    5503 generic.go:356] "Generic (PLEG): container finished" podID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerID="5e4434af2019e3c60fdd50d4266db6bc5fe6b73785c692f0ff5fc8085018c778" exitCode=60
Oct 30 22:45:05.067664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.067573    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" event={"ID":"128f1827-884b-4a6e-bc52-ef41d84ddbc0","Type":"ContainerDied","Data":"5e4434af2019e3c60fdd50d4266db6bc5fe6b73785c692f0ff5fc8085018c778"}
Oct 30 22:45:05.067850 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.067831    5503 scope.go:122] "RemoveContainer" containerID="5e4434af2019e3c60fdd50d4266db6bc5fe6b73785c692f0ff5fc8085018c778"
Oct 30 22:45:05.068056 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.068037    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv_odh-ai-gateway-infra(128f1827-884b-4a6e-bc52-ef41d84ddbc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0"
Oct 30 22:45:05.069480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.069461    5503 generic.go:356] "Generic (PLEG): container finished" podID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerID="f3db9d284c80b8aa02357ce97457b16ff522d38ec1fa0d8b2a5d1049b0783539" exitCode=60
Oct 30 22:45:05.069561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.069528    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" event={"ID":"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6","Type":"ContainerDied","Data":"f3db9d284c80b8aa02357ce97457b16ff522d38ec1fa0d8b2a5d1049b0783539"}
Oct 30 22:45:05.069711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.069697    5503 scope.go:122] "RemoveContainer" containerID="f3db9d284c80b8aa02357ce97457b16ff522d38ec1fa0d8b2a5d1049b0783539"
Oct 30 22:45:05.069871 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:05.069856    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp_odh-ai-gateway-infra(40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6"
Oct 30 22:45:05.078503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.078483    5503 scope.go:122] "RemoveContainer" containerID="439f2162207bd15593a54b2e679335eec17b4ac07b9a671fe682b9af6999c97f"
Oct 30 22:45:05.090894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.090877    5503 scope.go:122] "RemoveContainer" containerID="41bcb027998e226830fc4af1382d6e42c35f338ad8fa73fe195702c94cd166cc"
Oct 30 22:45:05.103602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.103583    5503 scope.go:122] "RemoveContainer" containerID="251c4798cdededb95e080c2a5c59f17274aafdb200d10cb2d0f76248227d12ae"
Oct 30 22:45:05.116995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.116976    5503 scope.go:122] "RemoveContainer" containerID="b3303235443a6319b3ccde9bdfcf641bf6f257cef75bb813e1520327ec620e1c"
Oct 30 22:45:05.129444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.129425    5503 scope.go:122] "RemoveContainer" containerID="019fe0a451ce03fad5c4499d2525b8b88ef30d814e0e9379d6b2c65b0716652b"
Oct 30 22:45:05.141856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.141840    5503 scope.go:122] "RemoveContainer" containerID="aadcc12e0b4f213a880b27fb8d333697502f4aa422dcae90fde8413de4e59220"
Oct 30 22:45:05.154142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.154125    5503 scope.go:122] "RemoveContainer" containerID="4ca9543c7a6c372d585b619be49bb6d42c9da74c4d6a771dda0138eba7385f52"
Oct 30 22:45:05.167090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.167073    5503 scope.go:122] "RemoveContainer" containerID="a4bc094a9520f93550870a67c7f73dc22f545921955d8bbdd027907c3da434b9"
Oct 30 22:45:05.351005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.350955    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" podStartSLOduration=5.3509411270000005 podStartE2EDuration="5.350941127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.349784384 +0000 UTC m=+3481.684361153" watchObservedRunningTime="2026-10-30 22:45:05.350941127 +0000 UTC m=+3481.685517896"
Oct 30 22:45:05.805521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:05.805491    5503 scope.go:122] "RemoveContainer" containerID="456565cb40635acf0bee437b89d6ca0634fadd9ea5746ccf113ddea0d1ff29eb"
Oct 30 22:45:06.078367 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.078287    5503 scope.go:122] "RemoveContainer" containerID="5764fe52f8895a521e61169ed9004b6340901b6a571e3da88fe2301e248887fc"
Oct 30 22:45:06.078829 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.078597    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d_odh-ai-gateway-infra(399b48c0-01d3-47b7-8238-4b1b7f650212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212"
Oct 30 22:45:06.079945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.079921    5503 scope.go:122] "RemoveContainer" containerID="5850fb343ebc28994b4901dbded9a5e0146f85fcbd377cbb4c19ed6e6f4a4068"
Oct 30 22:45:06.080196 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.080175    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-h7rxm_odh-ai-gateway-infra(942a698d-acea-43ab-9f76-f4b3e920a8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1"
Oct 30 22:45:06.081450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.081429    5503 scope.go:122] "RemoveContainer" containerID="4b895377c34c8c387b12cc53d71b46830c1a372a59eab512dd568d8e7e16dfb0"
Oct 30 22:45:06.081767 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.081739    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw_odh-ai-gateway-infra(d649e45e-f885-42d9-82e8-4b03d3ed144d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d"
Oct 30 22:45:06.083081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.083063    5503 scope.go:122] "RemoveContainer" containerID="83d236eb81ace936cbf12d71d1b543ce4d2f3668cea10ca442e5a538fb777f40"
Oct 30 22:45:06.083300 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.083282    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5_odh-ai-gateway-infra(d79579bf-99bc-4260-a73e-061c181cb66a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" podUID="d79579bf-99bc-4260-a73e-061c181cb66a"
Oct 30 22:45:06.084591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.084578    5503 scope.go:122] "RemoveContainer" containerID="bcc4b2b13ef9ce068dc1d609fb8076774b576a8206693f2d31272ae8642241d0"
Oct 30 22:45:06.084855 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.084829    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp_odh-ai-gateway-infra(33919e28-59ee-4a39-884b-59dc335abf8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" podUID="33919e28-59ee-4a39-884b-59dc335abf8c"
Oct 30 22:45:06.086126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.086107    5503 scope.go:122] "RemoveContainer" containerID="83e20e8b19181319f5669e9f85642a4929208c3533ef37a33147bcb840049e65"
Oct 30 22:45:06.086317 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.086295    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp_odh-ai-gateway-infra(0fc345c1-a266-4e1d-a688-ec82e72cf4e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5"
Oct 30 22:45:06.087705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.087685    5503 scope.go:122] "RemoveContainer" containerID="3dce0ab346e6af3210c1f95c8f6ec6d04a9b29ffad93f05135d054965a69dbb6"
Oct 30 22:45:06.087899 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.087875    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn_odh-ai-gateway-infra(c245e74f-8e3a-4898-9591-6ee505db9c7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d"
Oct 30 22:45:06.089235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.089218    5503 scope.go:122] "RemoveContainer" containerID="5e4434af2019e3c60fdd50d4266db6bc5fe6b73785c692f0ff5fc8085018c778"
Oct 30 22:45:06.089447 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.089425    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv_odh-ai-gateway-infra(128f1827-884b-4a6e-bc52-ef41d84ddbc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0"
Oct 30 22:45:06.090968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.090952    5503 scope.go:122] "RemoveContainer" containerID="f3db9d284c80b8aa02357ce97457b16ff522d38ec1fa0d8b2a5d1049b0783539"
Oct 30 22:45:06.091141 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.091123    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp_odh-ai-gateway-infra(40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6"
Oct 30 22:45:06.092398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:06.092381    5503 scope.go:122] "RemoveContainer" containerID="739589ea4525d2cdbb497cea411b98e1b1d0d1a068dd334576362b328c09ef09"
Oct 30 22:45:06.092563 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:06.092546    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h_odh-ai-gateway-infra(2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7"
Oct 30 22:45:08.687477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:08.687440    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:08.845250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:08.845217    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:08.996682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:08.996568    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:10.332901 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:10.330332    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:45:13.665374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:13.665337    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:14.127054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:14.126946    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:14.783001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:14.782967    5503 scope.go:122] "RemoveContainer" containerID="5e4434af2019e3c60fdd50d4266db6bc5fe6b73785c692f0ff5fc8085018c778"
Oct 30 22:45:14.783381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:14.783072    5503 scope.go:122] "RemoveContainer" containerID="f3db9d284c80b8aa02357ce97457b16ff522d38ec1fa0d8b2a5d1049b0783539"
Oct 30 22:45:15.479512 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:15.479458    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:15.782214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:15.782141    5503 scope.go:122] "RemoveContainer" containerID="5850fb343ebc28994b4901dbded9a5e0146f85fcbd377cbb4c19ed6e6f4a4068"
Oct 30 22:45:15.782370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:15.782252    5503 scope.go:122] "RemoveContainer" containerID="5764fe52f8895a521e61169ed9004b6340901b6a571e3da88fe2301e248887fc"
Oct 30 22:45:16.189974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.189055    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:16.292982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.292947    5503 generic.go:356] "Generic (PLEG): container finished" podID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerID="71895b3c53e0bdb8dc49c768837026ff28103e8bc941824ec4910a3eabc8bf28" exitCode=60
Oct 30 22:45:16.293273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.293251    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" event={"ID":"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6","Type":"ContainerDied","Data":"71895b3c53e0bdb8dc49c768837026ff28103e8bc941824ec4910a3eabc8bf28"}
Oct 30 22:45:16.293345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.293325    5503 scope.go:122] "RemoveContainer" containerID="f3db9d284c80b8aa02357ce97457b16ff522d38ec1fa0d8b2a5d1049b0783539"
Oct 30 22:45:16.293882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.293861    5503 scope.go:122] "RemoveContainer" containerID="71895b3c53e0bdb8dc49c768837026ff28103e8bc941824ec4910a3eabc8bf28"
Oct 30 22:45:16.294259 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:16.294233    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp_odh-ai-gateway-infra(40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6"
Oct 30 22:45:16.297254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.297232    5503 generic.go:356] "Generic (PLEG): container finished" podID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerID="f09b6298bfd620308317478be4d12fc0be76b1fb9b0b09cc87a66f8c1e36dd32" exitCode=60
Oct 30 22:45:16.297351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.297282    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" event={"ID":"128f1827-884b-4a6e-bc52-ef41d84ddbc0","Type":"ContainerDied","Data":"f09b6298bfd620308317478be4d12fc0be76b1fb9b0b09cc87a66f8c1e36dd32"}
Oct 30 22:45:16.297763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.297580    5503 scope.go:122] "RemoveContainer" containerID="f09b6298bfd620308317478be4d12fc0be76b1fb9b0b09cc87a66f8c1e36dd32"
Oct 30 22:45:16.297965 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:16.297908    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv_odh-ai-gateway-infra(128f1827-884b-4a6e-bc52-ef41d84ddbc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0"
Oct 30 22:45:16.310392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.310371    5503 scope.go:122] "RemoveContainer" containerID="5e4434af2019e3c60fdd50d4266db6bc5fe6b73785c692f0ff5fc8085018c778"
Oct 30 22:45:16.375096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.374190    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:16.407356 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:16.407317    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399b48c0_01d3_47b7_8238_4b1b7f650212.slice/crio-37b1e843c29461a27f0d4a032382bac79e5f5ed11738725608c7574416d6e192.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:45:16.782798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.782753    5503 scope.go:122] "RemoveContainer" containerID="739589ea4525d2cdbb497cea411b98e1b1d0d1a068dd334576362b328c09ef09"
Oct 30 22:45:16.782970 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.782913    5503 scope.go:122] "RemoveContainer" containerID="bcc4b2b13ef9ce068dc1d609fb8076774b576a8206693f2d31272ae8642241d0"
Oct 30 22:45:16.783055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.783037    5503 scope.go:122] "RemoveContainer" containerID="83d236eb81ace936cbf12d71d1b543ce4d2f3668cea10ca442e5a538fb777f40"
Oct 30 22:45:16.783169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:16.783103    5503 scope.go:122] "RemoveContainer" containerID="4b895377c34c8c387b12cc53d71b46830c1a372a59eab512dd568d8e7e16dfb0"
Oct 30 22:45:17.310470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.310385    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerStarted","Data":"8ccf178540ac8cd1998fd18a97f92a5263256935611ee113644b9ed291781834"}
Oct 30 22:45:17.312723 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.312690    5503 generic.go:356] "Generic (PLEG): container finished" podID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerID="37b1e843c29461a27f0d4a032382bac79e5f5ed11738725608c7574416d6e192" exitCode=60
Oct 30 22:45:17.312859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.312758    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" event={"ID":"399b48c0-01d3-47b7-8238-4b1b7f650212","Type":"ContainerDied","Data":"37b1e843c29461a27f0d4a032382bac79e5f5ed11738725608c7574416d6e192"}
Oct 30 22:45:17.312859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.312813    5503 scope.go:122] "RemoveContainer" containerID="5764fe52f8895a521e61169ed9004b6340901b6a571e3da88fe2301e248887fc"
Oct 30 22:45:17.313188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.313171    5503 scope.go:122] "RemoveContainer" containerID="37b1e843c29461a27f0d4a032382bac79e5f5ed11738725608c7574416d6e192"
Oct 30 22:45:17.313525 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:17.313505    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d_odh-ai-gateway-infra(399b48c0-01d3-47b7-8238-4b1b7f650212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212"
Oct 30 22:45:17.314993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.314970    5503 generic.go:356] "Generic (PLEG): container finished" podID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerID="86a0018906dc0f029131699ced97188a68384ea60d8711aebcd66557b7788933" exitCode=60
Oct 30 22:45:17.315112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.315008    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" event={"ID":"942a698d-acea-43ab-9f76-f4b3e920a8f1","Type":"ContainerDied","Data":"86a0018906dc0f029131699ced97188a68384ea60d8711aebcd66557b7788933"}
Oct 30 22:45:17.315501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.315409    5503 scope.go:122] "RemoveContainer" containerID="86a0018906dc0f029131699ced97188a68384ea60d8711aebcd66557b7788933"
Oct 30 22:45:17.315749 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:17.315727    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-h7rxm_odh-ai-gateway-infra(942a698d-acea-43ab-9f76-f4b3e920a8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1"
Oct 30 22:45:17.319584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.317694    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerStarted","Data":"1b4df7ce8a2fe0a3b1711b4b11a5d9e92d18c29fcb8da2f7abccb9f8874f77c4"}
Oct 30 22:45:17.333409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.333388    5503 scope.go:122] "RemoveContainer" containerID="5850fb343ebc28994b4901dbded9a5e0146f85fcbd377cbb4c19ed6e6f4a4068"
Oct 30 22:45:17.423032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.422990    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"]
Oct 30 22:45:17.544654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.544604    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"]
Oct 30 22:45:17.577590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.577565    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:17.643957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.643923    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-kube-api-access-894tr\" (UniqueName: \"kubernetes.io/projected/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-kube-api-access-894tr\") pod \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") "
Oct 30 22:45:17.644163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.644031    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-maas-api-service-ca\") pod \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\" (UID: \"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6\") "
Oct 30 22:45:17.644812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.644438    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-maas-api-service-ca" pod "40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" (UID: "40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:17.644812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.644550    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.646790 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.646756    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-kube-api-access-894tr" pod "40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" (UID: "40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6"). InnerVolumeSpecName "kube-api-access-894tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.691450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.691422    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:17.745571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.745537    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/128f1827-884b-4a6e-bc52-ef41d84ddbc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128f1827-884b-4a6e-bc52-ef41d84ddbc0-maas-api-service-ca\") pod \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") "
Oct 30 22:45:17.745767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.745649    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/128f1827-884b-4a6e-bc52-ef41d84ddbc0-kube-api-access-tjcl8\" (UniqueName: \"kubernetes.io/projected/128f1827-884b-4a6e-bc52-ef41d84ddbc0-kube-api-access-tjcl8\") pod \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\" (UID: \"128f1827-884b-4a6e-bc52-ef41d84ddbc0\") "
Oct 30 22:45:17.745945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.745918    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128f1827-884b-4a6e-bc52-ef41d84ddbc0-maas-api-service-ca" pod "128f1827-884b-4a6e-bc52-ef41d84ddbc0" (UID: "128f1827-884b-4a6e-bc52-ef41d84ddbc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:17.746086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.746072    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128f1827-884b-4a6e-bc52-ef41d84ddbc0-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.746144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.746089    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894tr\" (UniqueName: \"kubernetes.io/projected/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6-kube-api-access-894tr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.747682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.747660    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128f1827-884b-4a6e-bc52-ef41d84ddbc0-kube-api-access-tjcl8" pod "128f1827-884b-4a6e-bc52-ef41d84ddbc0" (UID: "128f1827-884b-4a6e-bc52-ef41d84ddbc0"). InnerVolumeSpecName "kube-api-access-tjcl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.782369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.782337    5503 scope.go:122] "RemoveContainer" containerID="3dce0ab346e6af3210c1f95c8f6ec6d04a9b29ffad93f05135d054965a69dbb6"
Oct 30 22:45:17.782571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.782454    5503 scope.go:122] "RemoveContainer" containerID="83e20e8b19181319f5669e9f85642a4929208c3533ef37a33147bcb840049e65"
Oct 30 22:45:17.782661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.782610    5503 scope.go:122] "RemoveContainer" containerID="73f93973749b2b9b98421e9230543d5c9c01124f9024e562bbfd429b0b2f6168"
Oct 30 22:45:17.848123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:17.848087    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjcl8\" (UniqueName: \"kubernetes.io/projected/128f1827-884b-4a6e-bc52-ef41d84ddbc0-kube-api-access-tjcl8\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.319501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.319297    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:18.331812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.331786    5503 generic.go:356] "Generic (PLEG): container finished" podID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerID="8ccf178540ac8cd1998fd18a97f92a5263256935611ee113644b9ed291781834" exitCode=60
Oct 30 22:45:18.331950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.331918    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerDied","Data":"8ccf178540ac8cd1998fd18a97f92a5263256935611ee113644b9ed291781834"}
Oct 30 22:45:18.332020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.331983    5503 scope.go:122] "RemoveContainer" containerID="739589ea4525d2cdbb497cea411b98e1b1d0d1a068dd334576362b328c09ef09"
Oct 30 22:45:18.332490 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.332428    5503 scope.go:122] "RemoveContainer" containerID="8ccf178540ac8cd1998fd18a97f92a5263256935611ee113644b9ed291781834"
Oct 30 22:45:18.332781 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:18.332760    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h_odh-ai-gateway-infra(2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7"
Oct 30 22:45:18.339567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.339539    5503 generic.go:356] "Generic (PLEG): container finished" podID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerID="0cdad90f649d8d144720a390b6a0c3c7c8cf1d01954484684391e13e1cdfc3f9" exitCode=60
Oct 30 22:45:18.339690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.339604    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" event={"ID":"d649e45e-f885-42d9-82e8-4b03d3ed144d","Type":"ContainerDied","Data":"0cdad90f649d8d144720a390b6a0c3c7c8cf1d01954484684391e13e1cdfc3f9"}
Oct 30 22:45:18.340148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.340054    5503 scope.go:122] "RemoveContainer" containerID="0cdad90f649d8d144720a390b6a0c3c7c8cf1d01954484684391e13e1cdfc3f9"
Oct 30 22:45:18.340382 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:18.340362    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw_odh-ai-gateway-infra(d649e45e-f885-42d9-82e8-4b03d3ed144d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d"
Oct 30 22:45:18.342033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.342014    5503 generic.go:356] "Generic (PLEG): container finished" podID="d79579bf-99bc-4260-a73e-061c181cb66a" containerID="1b4df7ce8a2fe0a3b1711b4b11a5d9e92d18c29fcb8da2f7abccb9f8874f77c4" exitCode=60
Oct 30 22:45:18.342135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.342089    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerDied","Data":"1b4df7ce8a2fe0a3b1711b4b11a5d9e92d18c29fcb8da2f7abccb9f8874f77c4"}
Oct 30 22:45:18.342458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.342438    5503 scope.go:122] "RemoveContainer" containerID="1b4df7ce8a2fe0a3b1711b4b11a5d9e92d18c29fcb8da2f7abccb9f8874f77c4"
Oct 30 22:45:18.342782 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:18.342759    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5_odh-ai-gateway-infra(d79579bf-99bc-4260-a73e-061c181cb66a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" podUID="d79579bf-99bc-4260-a73e-061c181cb66a"
Oct 30 22:45:18.343925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.343895    5503 generic.go:356] "Generic (PLEG): container finished" podID="33919e28-59ee-4a39-884b-59dc335abf8c" containerID="4e947d875bd557d8d3df9f5babc1aaae7ac8e7b68a963ca583fdffae9e3e44d6" exitCode=60
Oct 30 22:45:18.344023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.343978    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" event={"ID":"33919e28-59ee-4a39-884b-59dc335abf8c","Type":"ContainerDied","Data":"4e947d875bd557d8d3df9f5babc1aaae7ac8e7b68a963ca583fdffae9e3e44d6"}
Oct 30 22:45:18.344358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.344343    5503 scope.go:122] "RemoveContainer" containerID="4e947d875bd557d8d3df9f5babc1aaae7ac8e7b68a963ca583fdffae9e3e44d6"
Oct 30 22:45:18.344648 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:18.344612    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp_odh-ai-gateway-infra(33919e28-59ee-4a39-884b-59dc335abf8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" podUID="33919e28-59ee-4a39-884b-59dc335abf8c"
Oct 30 22:45:18.345967 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.345944    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" event={"ID":"c245e74f-8e3a-4898-9591-6ee505db9c7d","Type":"ContainerStarted","Data":"d1ede7812abd264a096f1c475837e1390adbc94f7a30daa53d21e7e89b5e6225"}
Oct 30 22:45:18.348273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.348254    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" event={"ID":"a50e8631-01a7-4c0d-9aa1-2fc504473f92","Type":"ContainerStarted","Data":"8f440ae32a7ded660873be849aa703d6ee056998a399f98852e41a72914a6239"}
Oct 30 22:45:18.348686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.348612    5503 scope.go:122] "RemoveContainer" containerID="4b895377c34c8c387b12cc53d71b46830c1a372a59eab512dd568d8e7e16dfb0"
Oct 30 22:45:18.349919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.349826    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"
Oct 30 22:45:18.349919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.349884    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv" event={"ID":"128f1827-884b-4a6e-bc52-ef41d84ddbc0","Type":"ContainerDied","Data":"76611fb28ed73621af7db5ad00f1caa7e5fa2d156e8f667a19a5d8e743472cf9"}
Oct 30 22:45:18.352058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.351897    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp" event={"ID":"40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6","Type":"ContainerDied","Data":"60b30e8d77b80d99efbf2130e8d70d7e36a4826cc281097ff73f24220657e99f"}
Oct 30 22:45:18.352163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.352073    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"
Oct 30 22:45:18.397681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.397618    5503 scope.go:122] "RemoveContainer" containerID="83d236eb81ace936cbf12d71d1b543ce4d2f3668cea10ca442e5a538fb777f40"
Oct 30 22:45:18.411868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.411848    5503 scope.go:122] "RemoveContainer" containerID="bcc4b2b13ef9ce068dc1d609fb8076774b576a8206693f2d31272ae8642241d0"
Oct 30 22:45:18.431915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.431888    5503 scope.go:122] "RemoveContainer" containerID="f09b6298bfd620308317478be4d12fc0be76b1fb9b0b09cc87a66f8c1e36dd32"
Oct 30 22:45:18.439965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.439934    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"]
Oct 30 22:45:18.444180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.444153    5503 scope.go:122] "RemoveContainer" containerID="71895b3c53e0bdb8dc49c768837026ff28103e8bc941824ec4910a3eabc8bf28"
Oct 30 22:45:18.561329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.561285    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"]
Oct 30 22:45:18.694284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.694247    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"]
Oct 30 22:45:18.699763 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.699735    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9717a8-29890005-l7fxv"]
Oct 30 22:45:18.727696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.727672    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:18.775607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.775574    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399b48c0-01d3-47b7-8238-4b1b7f650212-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/399b48c0-01d3-47b7-8238-4b1b7f650212-maas-api-service-ca\") pod \"399b48c0-01d3-47b7-8238-4b1b7f650212\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") "
Oct 30 22:45:18.775795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.775751    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399b48c0-01d3-47b7-8238-4b1b7f650212-kube-api-access-66r2t\" (UniqueName: \"kubernetes.io/projected/399b48c0-01d3-47b7-8238-4b1b7f650212-kube-api-access-66r2t\") pod \"399b48c0-01d3-47b7-8238-4b1b7f650212\" (UID: \"399b48c0-01d3-47b7-8238-4b1b7f650212\") "
Oct 30 22:45:18.776036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.776007    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399b48c0-01d3-47b7-8238-4b1b7f650212-maas-api-service-ca" pod "399b48c0-01d3-47b7-8238-4b1b7f650212" (UID: "399b48c0-01d3-47b7-8238-4b1b7f650212"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.776309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.776292    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/399b48c0-01d3-47b7-8238-4b1b7f650212-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.777885 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.777848    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399b48c0-01d3-47b7-8238-4b1b7f650212-kube-api-access-66r2t" pod "399b48c0-01d3-47b7-8238-4b1b7f650212" (UID: "399b48c0-01d3-47b7-8238-4b1b7f650212"). InnerVolumeSpecName "kube-api-access-66r2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.793115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.793086    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"]
Oct 30 22:45:18.877919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.877876    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66r2t\" (UniqueName: \"kubernetes.io/projected/399b48c0-01d3-47b7-8238-4b1b7f650212-kube-api-access-66r2t\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.918484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.918451    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"]
Oct 30 22:45:18.933208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.933175    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"]
Oct 30 22:45:18.942055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:18.942029    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e65019-29890005-csqmp"]
Oct 30 22:45:19.096033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.096007    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:19.180295 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.180253    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942a698d-acea-43ab-9f76-f4b3e920a8f1-kube-api-access-hc5wf\" (UniqueName: \"kubernetes.io/projected/942a698d-acea-43ab-9f76-f4b3e920a8f1-kube-api-access-hc5wf\") pod \"942a698d-acea-43ab-9f76-f4b3e920a8f1\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") "
Oct 30 22:45:19.180647 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.180603    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942a698d-acea-43ab-9f76-f4b3e920a8f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942a698d-acea-43ab-9f76-f4b3e920a8f1-maas-api-service-ca\") pod \"942a698d-acea-43ab-9f76-f4b3e920a8f1\" (UID: \"942a698d-acea-43ab-9f76-f4b3e920a8f1\") "
Oct 30 22:45:19.181001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.180976    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942a698d-acea-43ab-9f76-f4b3e920a8f1-maas-api-service-ca" pod "942a698d-acea-43ab-9f76-f4b3e920a8f1" (UID: "942a698d-acea-43ab-9f76-f4b3e920a8f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.182404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.182374    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942a698d-acea-43ab-9f76-f4b3e920a8f1-kube-api-access-hc5wf" pod "942a698d-acea-43ab-9f76-f4b3e920a8f1" (UID: "942a698d-acea-43ab-9f76-f4b3e920a8f1"). InnerVolumeSpecName "kube-api-access-hc5wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.283492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.283420    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc5wf\" (UniqueName: \"kubernetes.io/projected/942a698d-acea-43ab-9f76-f4b3e920a8f1-kube-api-access-hc5wf\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.283492 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.283448    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/942a698d-acea-43ab-9f76-f4b3e920a8f1-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.362131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.362099    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d" event={"ID":"399b48c0-01d3-47b7-8238-4b1b7f650212","Type":"ContainerDied","Data":"92988ca28cfa71887e4e194d15289d3ed28cde7836df12d21b0cd8162e040eeb"}
Oct 30 22:45:19.362651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.362112    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"
Oct 30 22:45:19.362789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.362621    5503 scope.go:122] "RemoveContainer" containerID="37b1e843c29461a27f0d4a032382bac79e5f5ed11738725608c7574416d6e192"
Oct 30 22:45:19.366245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.366220    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm" event={"ID":"942a698d-acea-43ab-9f76-f4b3e920a8f1","Type":"ContainerDied","Data":"a483b06ef471f50e4f1f9c5eda4d16afa381fa0d50f9b3b90187ff54dfe152cd"}
Oct 30 22:45:19.366411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.366226    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"
Oct 30 22:45:19.378422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.378388    5503 generic.go:356] "Generic (PLEG): container finished" podID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerID="4a0aaaba807d7381ac9f8e7879f8ab563295d80d672126b3dd4a8e90e6923d26" exitCode=60
Oct 30 22:45:19.378522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.378463    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" event={"ID":"0fc345c1-a266-4e1d-a688-ec82e72cf4e5","Type":"ContainerDied","Data":"4a0aaaba807d7381ac9f8e7879f8ab563295d80d672126b3dd4a8e90e6923d26"}
Oct 30 22:45:19.378990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.378973    5503 scope.go:122] "RemoveContainer" containerID="86a0018906dc0f029131699ced97188a68384ea60d8711aebcd66557b7788933"
Oct 30 22:45:19.379380 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.379363    5503 scope.go:122] "RemoveContainer" containerID="4a0aaaba807d7381ac9f8e7879f8ab563295d80d672126b3dd4a8e90e6923d26"
Oct 30 22:45:19.379796 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:19.379774    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp_odh-ai-gateway-infra(0fc345c1-a266-4e1d-a688-ec82e72cf4e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5"
Oct 30 22:45:19.383943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.383922    5503 generic.go:356] "Generic (PLEG): container finished" podID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerID="d1ede7812abd264a096f1c475837e1390adbc94f7a30daa53d21e7e89b5e6225" exitCode=60
Oct 30 22:45:19.384046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.384021    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" event={"ID":"c245e74f-8e3a-4898-9591-6ee505db9c7d","Type":"ContainerDied","Data":"d1ede7812abd264a096f1c475837e1390adbc94f7a30daa53d21e7e89b5e6225"}
Oct 30 22:45:19.384410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.384336    5503 scope.go:122] "RemoveContainer" containerID="d1ede7812abd264a096f1c475837e1390adbc94f7a30daa53d21e7e89b5e6225"
Oct 30 22:45:19.384684 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:19.384610    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn_odh-ai-gateway-infra(c245e74f-8e3a-4898-9591-6ee505db9c7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d"
Oct 30 22:45:19.386386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.386363    5503 generic.go:356] "Generic (PLEG): container finished" podID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerID="8f440ae32a7ded660873be849aa703d6ee056998a399f98852e41a72914a6239" exitCode=60
Oct 30 22:45:19.386460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.386445    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" event={"ID":"a50e8631-01a7-4c0d-9aa1-2fc504473f92","Type":"ContainerDied","Data":"8f440ae32a7ded660873be849aa703d6ee056998a399f98852e41a72914a6239"}
Oct 30 22:45:19.386798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.386782    5503 scope.go:122] "RemoveContainer" containerID="8f440ae32a7ded660873be849aa703d6ee056998a399f98852e41a72914a6239"
Oct 30 22:45:19.387043 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:19.387020    5503 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s_odh-ai-gateway-infra(a50e8631-01a7-4c0d-9aa1-2fc504473f92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92"
Oct 30 22:45:19.390547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.390530    5503 scope.go:122] "RemoveContainer" containerID="83e20e8b19181319f5669e9f85642a4929208c3533ef37a33147bcb840049e65"
Oct 30 22:45:19.418943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.418820    5503 scope.go:122] "RemoveContainer" containerID="3dce0ab346e6af3210c1f95c8f6ec6d04a9b29ffad93f05135d054965a69dbb6"
Oct 30 22:45:19.437562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.437431    5503 scope.go:122] "RemoveContainer" containerID="73f93973749b2b9b98421e9230543d5c9c01124f9024e562bbfd429b0b2f6168"
Oct 30 22:45:19.461673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.461184    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:19.556821 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.556796    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:19.565062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.565037    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:19.588491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.588463    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-kube-api-access-vjtj8\" (UniqueName: \"kubernetes.io/projected/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-kube-api-access-vjtj8\") pod \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") "
Oct 30 22:45:19.588619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.588601    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-maas-api-service-ca\") pod \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\" (UID: \"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7\") "
Oct 30 22:45:19.588923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.588897    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-maas-api-service-ca" pod "2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" (UID: "2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.589074 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.589061    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.590520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.590496    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-kube-api-access-vjtj8" pod "2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" (UID: "2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7"). InnerVolumeSpecName "kube-api-access-vjtj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.621399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.621373    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:19.652746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.652718    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"]
Oct 30 22:45:19.657188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.657160    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-37e79b-29890005-6fs9d"]
Oct 30 22:45:19.689731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.689699    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d79579bf-99bc-4260-a73e-061c181cb66a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79579bf-99bc-4260-a73e-061c181cb66a-maas-api-service-ca\") pod \"d79579bf-99bc-4260-a73e-061c181cb66a\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") "
Oct 30 22:45:19.689900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.689875    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d79579bf-99bc-4260-a73e-061c181cb66a-kube-api-access-lnwj9\" (UniqueName: \"kubernetes.io/projected/d79579bf-99bc-4260-a73e-061c181cb66a-kube-api-access-lnwj9\") pod \"d79579bf-99bc-4260-a73e-061c181cb66a\" (UID: \"d79579bf-99bc-4260-a73e-061c181cb66a\") "
Oct 30 22:45:19.690157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.690130    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79579bf-99bc-4260-a73e-061c181cb66a-maas-api-service-ca" pod "d79579bf-99bc-4260-a73e-061c181cb66a" (UID: "d79579bf-99bc-4260-a73e-061c181cb66a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.690353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.690337    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjtj8\" (UniqueName: \"kubernetes.io/projected/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7-kube-api-access-vjtj8\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.690395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.690357    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d79579bf-99bc-4260-a73e-061c181cb66a-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.692621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.692603    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79579bf-99bc-4260-a73e-061c181cb66a-kube-api-access-lnwj9" pod "d79579bf-99bc-4260-a73e-061c181cb66a" (UID: "d79579bf-99bc-4260-a73e-061c181cb66a"). InnerVolumeSpecName "kube-api-access-lnwj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.790593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.790558    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" path="/var/lib/kubelet/pods/128f1827-884b-4a6e-bc52-ef41d84ddbc0/volumes"
Oct 30 22:45:19.791050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.791024    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" path="/var/lib/kubelet/pods/399b48c0-01d3-47b7-8238-4b1b7f650212/volumes"
Oct 30 22:45:19.791439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.791415    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" path="/var/lib/kubelet/pods/40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6/volumes"
Oct 30 22:45:19.791622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.791602    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwj9\" (UniqueName: \"kubernetes.io/projected/d79579bf-99bc-4260-a73e-061c181cb66a-kube-api-access-lnwj9\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.826916 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:19.826886    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"]
Oct 30 22:45:20.008503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.008475    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:20.068118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.068035    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"]
Oct 30 22:45:20.094955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.094920    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33919e28-59ee-4a39-884b-59dc335abf8c-kube-api-access-qgsc5\" (UniqueName: \"kubernetes.io/projected/33919e28-59ee-4a39-884b-59dc335abf8c-kube-api-access-qgsc5\") pod \"33919e28-59ee-4a39-884b-59dc335abf8c\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") "
Oct 30 22:45:20.095103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.094978    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33919e28-59ee-4a39-884b-59dc335abf8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33919e28-59ee-4a39-884b-59dc335abf8c-maas-api-service-ca\") pod \"33919e28-59ee-4a39-884b-59dc335abf8c\" (UID: \"33919e28-59ee-4a39-884b-59dc335abf8c\") "
Oct 30 22:45:20.095306 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.095284    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33919e28-59ee-4a39-884b-59dc335abf8c-maas-api-service-ca" pod "33919e28-59ee-4a39-884b-59dc335abf8c" (UID: "33919e28-59ee-4a39-884b-59dc335abf8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.095508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.095496    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33919e28-59ee-4a39-884b-59dc335abf8c-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.097066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.097037    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33919e28-59ee-4a39-884b-59dc335abf8c-kube-api-access-qgsc5" pod "33919e28-59ee-4a39-884b-59dc335abf8c" (UID: "33919e28-59ee-4a39-884b-59dc335abf8c"). InnerVolumeSpecName "kube-api-access-qgsc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.122606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.122578    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"]
Oct 30 22:45:20.126296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.126271    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-h7rxm"]
Oct 30 22:45:20.184583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.184547    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"]
Oct 30 22:45:20.197109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.197079    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgsc5\" (UniqueName: \"kubernetes.io/projected/33919e28-59ee-4a39-884b-59dc335abf8c-kube-api-access-qgsc5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.304466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.304427    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"]
Oct 30 22:45:20.405120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.405086    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"
Oct 30 22:45:20.405120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.405107    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h" event={"ID":"2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7","Type":"ContainerDied","Data":"4d612c03cc22b8c44c9334e9397c50f238508f8f14a63629b6b31cfba56e0161"}
Oct 30 22:45:20.405684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.405172    5503 scope.go:122] "RemoveContainer" containerID="8ccf178540ac8cd1998fd18a97f92a5263256935611ee113644b9ed291781834"
Oct 30 22:45:20.411386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.411351    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5" event={"ID":"d79579bf-99bc-4260-a73e-061c181cb66a","Type":"ContainerDied","Data":"5f0dd7fcaab0cae2ae97c764a1d7db8f13cf931d485449417c0da68f65a47be8"}
Oct 30 22:45:20.420393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.420374    5503 scope.go:122] "RemoveContainer" containerID="1b4df7ce8a2fe0a3b1711b4b11a5d9e92d18c29fcb8da2f7abccb9f8874f77c4"
Oct 30 22:45:20.420711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.420688    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"
Oct 30 22:45:20.425861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.425836    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp" event={"ID":"33919e28-59ee-4a39-884b-59dc335abf8c","Type":"ContainerDied","Data":"2a14e7e1bb7d490293db113222b13aa3fe4b19f2caa26f8fe68fd607d723c6b9"}
Oct 30 22:45:20.425962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.425941    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"
Oct 30 22:45:20.429930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.429909    5503 scope.go:122] "RemoveContainer" containerID="4e947d875bd557d8d3df9f5babc1aaae7ac8e7b68a963ca583fdffae9e3e44d6"
Oct 30 22:45:20.485160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.485127    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:20.595463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.595433    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:20.602792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.602757    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d649e45e-f885-42d9-82e8-4b03d3ed144d-kube-api-access-tn8mv\" (UniqueName: \"kubernetes.io/projected/d649e45e-f885-42d9-82e8-4b03d3ed144d-kube-api-access-tn8mv\") pod \"d649e45e-f885-42d9-82e8-4b03d3ed144d\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") "
Oct 30 22:45:20.602959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.602929    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d649e45e-f885-42d9-82e8-4b03d3ed144d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d649e45e-f885-42d9-82e8-4b03d3ed144d-maas-api-service-ca\") pod \"d649e45e-f885-42d9-82e8-4b03d3ed144d\" (UID: \"d649e45e-f885-42d9-82e8-4b03d3ed144d\") "
Oct 30 22:45:20.603696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.603544    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d649e45e-f885-42d9-82e8-4b03d3ed144d-maas-api-service-ca" pod "d649e45e-f885-42d9-82e8-4b03d3ed144d" (UID: "d649e45e-f885-42d9-82e8-4b03d3ed144d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.606900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.606870    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d649e45e-f885-42d9-82e8-4b03d3ed144d-kube-api-access-tn8mv" pod "d649e45e-f885-42d9-82e8-4b03d3ed144d" (UID: "d649e45e-f885-42d9-82e8-4b03d3ed144d"). InnerVolumeSpecName "kube-api-access-tn8mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.633405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.633382    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:20.703917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.703884    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a50e8631-01a7-4c0d-9aa1-2fc504473f92-kube-api-access-bzv89\" (UniqueName: \"kubernetes.io/projected/a50e8631-01a7-4c0d-9aa1-2fc504473f92-kube-api-access-bzv89\") pod \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") "
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704048    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a50e8631-01a7-4c0d-9aa1-2fc504473f92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e8631-01a7-4c0d-9aa1-2fc504473f92-maas-api-service-ca\") pod \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\" (UID: \"a50e8631-01a7-4c0d-9aa1-2fc504473f92\") "
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704109    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c245e74f-8e3a-4898-9591-6ee505db9c7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c245e74f-8e3a-4898-9591-6ee505db9c7d-maas-api-service-ca\") pod \"c245e74f-8e3a-4898-9591-6ee505db9c7d\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") "
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704187    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c245e74f-8e3a-4898-9591-6ee505db9c7d-kube-api-access-v2n8z\" (UniqueName: \"kubernetes.io/projected/c245e74f-8e3a-4898-9591-6ee505db9c7d-kube-api-access-v2n8z\") pod \"c245e74f-8e3a-4898-9591-6ee505db9c7d\" (UID: \"c245e74f-8e3a-4898-9591-6ee505db9c7d\") "
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704446    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a50e8631-01a7-4c0d-9aa1-2fc504473f92-maas-api-service-ca" pod "a50e8631-01a7-4c0d-9aa1-2fc504473f92" (UID: "a50e8631-01a7-4c0d-9aa1-2fc504473f92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704448    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c245e74f-8e3a-4898-9591-6ee505db9c7d-maas-api-service-ca" pod "c245e74f-8e3a-4898-9591-6ee505db9c7d" (UID: "c245e74f-8e3a-4898-9591-6ee505db9c7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704624    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d649e45e-f885-42d9-82e8-4b03d3ed144d-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704659    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn8mv\" (UniqueName: \"kubernetes.io/projected/d649e45e-f885-42d9-82e8-4b03d3ed144d-kube-api-access-tn8mv\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704671    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a50e8631-01a7-4c0d-9aa1-2fc504473f92-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.704680    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c245e74f-8e3a-4898-9591-6ee505db9c7d-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.706088    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a50e8631-01a7-4c0d-9aa1-2fc504473f92-kube-api-access-bzv89" pod "a50e8631-01a7-4c0d-9aa1-2fc504473f92" (UID: "a50e8631-01a7-4c0d-9aa1-2fc504473f92"). InnerVolumeSpecName "kube-api-access-bzv89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.706198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.706153    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c245e74f-8e3a-4898-9591-6ee505db9c7d-kube-api-access-v2n8z" pod "c245e74f-8e3a-4898-9591-6ee505db9c7d" (UID: "c245e74f-8e3a-4898-9591-6ee505db9c7d"). InnerVolumeSpecName "kube-api-access-v2n8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.805403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.805371    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2n8z\" (UniqueName: \"kubernetes.io/projected/c245e74f-8e3a-4898-9591-6ee505db9c7d-kube-api-access-v2n8z\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.805403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.805396    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzv89\" (UniqueName: \"kubernetes.io/projected/a50e8631-01a7-4c0d-9aa1-2fc504473f92-kube-api-access-bzv89\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.945104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.945067    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"]
Oct 30 22:45:20.949078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:20.949054    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e524ce-29890005-rrctp"]
Oct 30 22:45:21.251229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.251141    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"]
Oct 30 22:45:21.305867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.305831    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:21.418258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.418227    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:21.419251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.419235    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"]
Oct 30 22:45:21.423611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.423584    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3b392d-29890005-krp7h"]
Oct 30 22:45:21.443751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.443714    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s" event={"ID":"a50e8631-01a7-4c0d-9aa1-2fc504473f92","Type":"ContainerDied","Data":"49b8ac58293541871f439781e16049da210c38cc33fd4d448a8e7165e53d91b5"}
Oct 30 22:45:21.443918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.443782    5503 scope.go:122] "RemoveContainer" containerID="8f440ae32a7ded660873be849aa703d6ee056998a399f98852e41a72914a6239"
Oct 30 22:45:21.443918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.443725    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"
Oct 30 22:45:21.445815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.445791    5503 generic.go:356] "Generic (PLEG): container finished" podID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerID="a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2" exitCode=0
Oct 30 22:45:21.445947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.445868    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" event={"ID":"c163a492-0c0a-4664-805b-5798e4cd42a2","Type":"ContainerDied","Data":"a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2"}
Oct 30 22:45:21.446316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.446284    5503 scope.go:122] "RemoveContainer" containerID="a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2"
Oct 30 22:45:21.449540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.448432    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:45:21.449540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.448519    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:45:21.449540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.448569    5503 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 22:45:21.455561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.453818    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw" event={"ID":"d649e45e-f885-42d9-82e8-4b03d3ed144d","Type":"ContainerDied","Data":"2b693605c01db9a2cb707e436cd31266462816c34975053b027be1ae4bcdf7eb"}
Oct 30 22:45:21.455561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.453840    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"
Oct 30 22:45:21.459795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.459776    5503 scope.go:122] "RemoveContainer" containerID="0cdad90f649d8d144720a390b6a0c3c7c8cf1d01954484684391e13e1cdfc3f9"
Oct 30 22:45:21.463132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.463114    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"
Oct 30 22:45:21.463247 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.463151    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp" event={"ID":"0fc345c1-a266-4e1d-a688-ec82e72cf4e5","Type":"ContainerDied","Data":"4a0c6b6ea517c735818ef215939de99ae296ebaeebf0631fcf615a7c332a4933"}
Oct 30 22:45:21.464709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.464681    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn" event={"ID":"c245e74f-8e3a-4898-9591-6ee505db9c7d","Type":"ContainerDied","Data":"c085908bcb258488ff8c528e64086ac61c6cccc32c2b45f6d30a63eddb7adc0b"}
Oct 30 22:45:21.464809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.464746    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"
Oct 30 22:45:21.472373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.472354    5503 scope.go:122] "RemoveContainer" containerID="4a0aaaba807d7381ac9f8e7879f8ab563295d80d672126b3dd4a8e90e6923d26"
Oct 30 22:45:21.489974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.489953    5503 scope.go:122] "RemoveContainer" containerID="d1ede7812abd264a096f1c475837e1390adbc94f7a30daa53d21e7e89b5e6225"
Oct 30 22:45:21.511507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.511482    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:21.513977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.513956    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-kube-api-access-67g57\" (UniqueName: \"kubernetes.io/projected/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-kube-api-access-67g57\") pod \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") "
Oct 30 22:45:21.514105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.514092    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-maas-api-service-ca\") pod \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\" (UID: \"0fc345c1-a266-4e1d-a688-ec82e72cf4e5\") "
Oct 30 22:45:21.514415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.514388    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-maas-api-service-ca" pod "0fc345c1-a266-4e1d-a688-ec82e72cf4e5" (UID: "0fc345c1-a266-4e1d-a688-ec82e72cf4e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.514569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.514549    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-maas-api-service-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:21.516156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.516121    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-kube-api-access-67g57" pod "0fc345c1-a266-4e1d-a688-ec82e72cf4e5" (UID: "0fc345c1-a266-4e1d-a688-ec82e72cf4e5"). InnerVolumeSpecName "kube-api-access-67g57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.536373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.536349    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"]
Oct 30 22:45:21.542401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.542374    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-435923-29890005-9tgn5"]
Oct 30 22:45:21.616115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.616082    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67g57\" (UniqueName: \"kubernetes.io/projected/0fc345c1-a266-4e1d-a688-ec82e72cf4e5-kube-api-access-67g57\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:21.663502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.663468    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"]
Oct 30 22:45:21.667986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.667959    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-74bb81-29890005-ncvwn"]
Oct 30 22:45:21.717288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.717253    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:21.779569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.779541    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"]
Oct 30 22:45:21.790033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.790006    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" path="/var/lib/kubelet/pods/2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7/volumes"
Oct 30 22:45:21.790414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.790401    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" path="/var/lib/kubelet/pods/33919e28-59ee-4a39-884b-59dc335abf8c/volumes"
Oct 30 22:45:21.790795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.790780    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" path="/var/lib/kubelet/pods/942a698d-acea-43ab-9f76-f4b3e920a8f1/volumes"
Oct 30 22:45:21.791439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.791424    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" path="/var/lib/kubelet/pods/c245e74f-8e3a-4898-9591-6ee505db9c7d/volumes"
Oct 30 22:45:21.791831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.791818    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" path="/var/lib/kubelet/pods/d79579bf-99bc-4260-a73e-061c181cb66a/volumes"
Oct 30 22:45:21.792122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.792109    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-68f80c-29890005-b2z8s"]
Oct 30 22:45:21.904403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.904364    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"]
Oct 30 22:45:21.906698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:21.906677    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fc6466-29890005-jvhlw"]
Oct 30 22:45:22.285384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:22.285347    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"]
Oct 30 22:45:22.288272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:22.288247    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d09fff-29890005-b5mkp"]
Oct 30 22:45:22.476833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:22.476798    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" event={"ID":"c163a492-0c0a-4664-805b-5798e4cd42a2","Type":"ContainerStarted","Data":"02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6"}
Oct 30 22:45:22.813991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:22.813958    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:45:22.814197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:22.814140    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:45:22.814278 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:22.814209    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 22:45:23.305436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:23.305391    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:23.790148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:23.790113    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" path="/var/lib/kubelet/pods/0fc345c1-a266-4e1d-a688-ec82e72cf4e5/volumes"
Oct 30 22:45:23.790576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:23.790560    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" path="/var/lib/kubelet/pods/a50e8631-01a7-4c0d-9aa1-2fc504473f92/volumes"
Oct 30 22:45:23.791002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:23.790988    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" path="/var/lib/kubelet/pods/d649e45e-f885-42d9-82e8-4b03d3ed144d/volumes"
Oct 30 22:45:23.814122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:23.814093    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:45:23.814274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:23.814148    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 22:45:24.563082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.563048    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:24.754665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.747948    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:24.813746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.813629    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:45:24.813746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.813718    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 22:45:24.977297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977263    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"]
Oct 30 22:45:24.977801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977786    5503 cpu_manager.go:414] "containerMap: removing container" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:24.977801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977801    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977813    5503 cpu_manager.go:414] "containerMap: removing container" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977819    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977829    5503 cpu_manager.go:414] "containerMap: removing container" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977835    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977844    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977849    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977855    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977859    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977866    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977871    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977877    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977882    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977888    5503 cpu_manager.go:414] "containerMap: removing container" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977893    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977899    5503 cpu_manager.go:414] "containerMap: removing container" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977903    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977908    5503 cpu_manager.go:414] "containerMap: removing container" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:24.977904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977912    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977920    5503 cpu_manager.go:414] "containerMap: removing container" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977924    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977931    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977937    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977942    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977947    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977954    5503 cpu_manager.go:414] "containerMap: removing container" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977958    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977964    5503 cpu_manager.go:414] "containerMap: removing container" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977968    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977974    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977979    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977985    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977989    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977995    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.977999    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978007    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978012    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978018    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978022    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978029    5503 cpu_manager.go:414] "containerMap: removing container" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978033    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978041    5503 cpu_manager.go:414] "containerMap: removing container" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978046    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978052    5503 cpu_manager.go:414] "containerMap: removing container" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978056    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978065    5503 cpu_manager.go:414] "containerMap: removing container" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978069    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978079    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978084    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978090    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978095    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978101    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978105    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978116    5503 cpu_manager.go:414] "containerMap: removing container" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978120    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978126    5503 cpu_manager.go:414] "containerMap: removing container" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978131    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978206    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978214    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978221    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978228    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978234    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978238    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978244    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978250    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978255    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978260    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978265    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978270    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d649e45e-f885-42d9-82e8-4b03d3ed144d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978275    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978280    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978285    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978290    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978296    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978301    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978307    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978312    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978317    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc8ac2f-dd6e-40c8-8f1d-d8e4207000b7" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978321    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978327    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978332    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978336    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978341    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="399b48c0-01d3-47b7-8238-4b1b7f650212" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978348    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:24.978533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.978352    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="128f1827-884b-4a6e-bc52-ef41d84ddbc0" containerName="cleanup"
Oct 30 22:45:24.982938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:24.982914    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.041659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.041510    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"]
Oct 30 22:45:25.088087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.087977    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b3f91-b0e5-4837-afce-0904cd9a7732-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.105658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.092163    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6sd\" (UniqueName: \"kubernetes.io/projected/187b3f91-b0e5-4837-afce-0904cd9a7732-kube-api-access-bd6sd\") pod \"payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.111197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.094874    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f4e241-dockercfg-xtvj8\""
Oct 30 22:45:25.111197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.094917    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f4e241\""
Oct 30 22:45:25.179430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.179391    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"]
Oct 30 22:45:25.180266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180245    5503 cpu_manager.go:414] "containerMap: removing container" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:25.180352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180268    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942a698d-acea-43ab-9f76-f4b3e920a8f1" containerName="cleanup"
Oct 30 22:45:25.180352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180304    5503 cpu_manager.go:414] "containerMap: removing container" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:25.180352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180312    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:25.180352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180338    5503 cpu_manager.go:414] "containerMap: removing container" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:25.180352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180347    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c245e74f-8e3a-4898-9591-6ee505db9c7d" containerName="cleanup"
Oct 30 22:45:25.180586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180385    5503 cpu_manager.go:414] "containerMap: removing container" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:25.180586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180393    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:25.180586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180515    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="33919e28-59ee-4a39-884b-59dc335abf8c" containerName="cleanup"
Oct 30 22:45:25.180586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180527    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b6b80a-1e19-4a4c-a2c2-9c180f03c2f6" containerName="cleanup"
Oct 30 22:45:25.180586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180537    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc345c1-a266-4e1d-a688-ec82e72cf4e5" containerName="cleanup"
Oct 30 22:45:25.180586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180546    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d79579bf-99bc-4260-a73e-061c181cb66a" containerName="cleanup"
Oct 30 22:45:25.180586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.180551    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a50e8631-01a7-4c0d-9aa1-2fc504473f92" containerName="cleanup"
Oct 30 22:45:25.184045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.184027    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.194734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.194698    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"]
Oct 30 22:45:25.213601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.211687    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b3f91-b0e5-4837-afce-0904cd9a7732-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.213601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.211785    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tbfd\" (UniqueName: \"kubernetes.io/projected/74a6a7fe-602d-4340-ab2c-1259b445e22f-kube-api-access-6tbfd\") pod \"payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.213601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.211845    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6sd\" (UniqueName: \"kubernetes.io/projected/187b3f91-b0e5-4837-afce-0904cd9a7732-kube-api-access-bd6sd\") pod \"payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.213601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.211895    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a6a7fe-602d-4340-ab2c-1259b445e22f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.213601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.212761    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b3f91-b0e5-4837-afce-0904cd9a7732-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.275069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.275037    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6sd\" (UniqueName: \"kubernetes.io/projected/187b3f91-b0e5-4837-afce-0904cd9a7732-kube-api-access-bd6sd\") pod \"payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.296558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.296533    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:25.312727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.312698    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tbfd\" (UniqueName: \"kubernetes.io/projected/74a6a7fe-602d-4340-ab2c-1259b445e22f-kube-api-access-6tbfd\") pod \"payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.312841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.312808    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a6a7fe-602d-4340-ab2c-1259b445e22f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.313466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.313440    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a6a7fe-602d-4340-ab2c-1259b445e22f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.375532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.375454    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tbfd\" (UniqueName: \"kubernetes.io/projected/74a6a7fe-602d-4340-ab2c-1259b445e22f-kube-api-access-6tbfd\") pod \"payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.538109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.536864    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:25.702144 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:25.702104    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187b3f91_b0e5_4837_afce_0904cd9a7732.slice/crio-421f1f5f763e8062f234a89d8b307b10dfe342029b6cce60017231ed4e8915f2 WatchSource:0}: Error finding container 421f1f5f763e8062f234a89d8b307b10dfe342029b6cce60017231ed4e8915f2: Status 404 returned error can't find the container with id 421f1f5f763e8062f234a89d8b307b10dfe342029b6cce60017231ed4e8915f2
Oct 30 22:45:25.705335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.705306    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"]
Oct 30 22:45:25.813923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.813895    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:45:25.814262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.813946    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 22:45:25.913008 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:25.912978    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a6a7fe_602d_4340_ab2c_1259b445e22f.slice/crio-689561ec0ca732b29a36c803fb232a10fe2bf972fc2ddc8facc9487604ec6c3b WatchSource:0}: Error finding container 689561ec0ca732b29a36c803fb232a10fe2bf972fc2ddc8facc9487604ec6c3b: Status 404 returned error can't find the container with id 689561ec0ca732b29a36c803fb232a10fe2bf972fc2ddc8facc9487604ec6c3b
Oct 30 22:45:25.923657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:25.916037    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"]
Oct 30 22:45:26.507491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.507459    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv" event={"ID":"74a6a7fe-602d-4340-ab2c-1259b445e22f","Type":"ContainerStarted","Data":"4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46"}
Oct 30 22:45:26.507716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.507509    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv" event={"ID":"74a6a7fe-602d-4340-ab2c-1259b445e22f","Type":"ContainerStarted","Data":"689561ec0ca732b29a36c803fb232a10fe2bf972fc2ddc8facc9487604ec6c3b"}
Oct 30 22:45:26.507716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.507557    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:26.509179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.509156    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt" event={"ID":"187b3f91-b0e5-4837-afce-0904cd9a7732","Type":"ContainerStarted","Data":"a54f68442deed2ef921eed04bc5e37a6829c76301231dbac2f6a9056b3278cb4"}
Oct 30 22:45:26.509304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.509195    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt" event={"ID":"187b3f91-b0e5-4837-afce-0904cd9a7732","Type":"ContainerStarted","Data":"421f1f5f763e8062f234a89d8b307b10dfe342029b6cce60017231ed4e8915f2"}
Oct 30 22:45:26.509304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.509240    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:26.635659 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.635242    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv" podStartSLOduration=1.635220831 podStartE2EDuration="1.635220831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:26.630889608 +0000 UTC m=+3502.965466379" watchObservedRunningTime="2026-10-30 22:45:26.635220831 +0000 UTC m=+3502.969797603"
Oct 30 22:45:26.814178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.814096    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 22:45:26.814178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:26.814151    5503 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 22:45:27.756059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:27.756022    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:27.818962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:27.818932    5503 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:45:27.819436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:27.819417    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:45:27.820182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:27.820164    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:45:27.904848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:27.904813    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:27.995075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:27.995015    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt" podStartSLOduration=3.99500047 podStartE2EDuration="3.99500047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:26.762781197 +0000 UTC m=+3503.097357967" watchObservedRunningTime="2026-10-30 22:45:27.99500047 +0000 UTC m=+3504.329577276"
Oct 30 22:45:28.098440 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:28.098356    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:45:28.507138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:28.507102    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:29.601654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:29.601606    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:30.823360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:30.823329    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:33.029073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:33.029039    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:37.519401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:37.519370    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:45:37.519948 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:37.519442    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:45:37.602675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:37.602610    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:39.169522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:39.169484    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:40.822057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:40.822019    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:41.682500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:41.682467    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:41.892841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:41.880767    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:42.124921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:42.124824    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.228800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:46.228752    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.427833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:46.427777    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.588873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:46.588788    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:46.830836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:46.830802    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:50.600115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:50.600066    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:50.752504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:50.752472    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:51.309663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.309603    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"]
Oct 30 22:45:51.312963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.312945    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.367995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.367965    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"]
Oct 30 22:45:51.416327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.416295    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-031eee\""
Oct 30 22:45:51.416327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.416300    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-031eee-dockercfg-4tdv9\""
Oct 30 22:45:51.459501 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.459465    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c991ba-8682-46ce-8b06-10af066a8e4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.459687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.459601    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6b77\" (UniqueName: \"kubernetes.io/projected/71c991ba-8682-46ce-8b06-10af066a8e4e-kube-api-access-h6b77\") pod \"payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.531714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.531681    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:51.561167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.561056    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c991ba-8682-46ce-8b06-10af066a8e4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.561167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.561142    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6b77\" (UniqueName: \"kubernetes.io/projected/71c991ba-8682-46ce-8b06-10af066a8e4e-kube-api-access-h6b77\") pod \"payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.561848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.561821    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c991ba-8682-46ce-8b06-10af066a8e4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.625283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.625143    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6b77\" (UniqueName: \"kubernetes.io/projected/71c991ba-8682-46ce-8b06-10af066a8e4e-kube-api-access-h6b77\") pod \"payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.627971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.627934    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:51.703779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.703705    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:51.852771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:51.852680    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:52.000997 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:45:52.000964    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c991ba_8682_46ce_8b06_10af066a8e4e.slice/crio-ba27eb6f3183fa0aec72cc2017fa6f27aa8c34d72c133d97c3f3935b463e5e48 WatchSource:0}: Error finding container ba27eb6f3183fa0aec72cc2017fa6f27aa8c34d72c133d97c3f3935b463e5e48: Status 404 returned error can't find the container with id ba27eb6f3183fa0aec72cc2017fa6f27aa8c34d72c133d97c3f3935b463e5e48
Oct 30 22:45:52.004137 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:52.004113    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"]
Oct 30 22:45:52.714528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:52.714489    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65" event={"ID":"71c991ba-8682-46ce-8b06-10af066a8e4e","Type":"ContainerStarted","Data":"a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600"}
Oct 30 22:45:52.715041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:52.714548    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65" event={"ID":"71c991ba-8682-46ce-8b06-10af066a8e4e","Type":"ContainerStarted","Data":"ba27eb6f3183fa0aec72cc2017fa6f27aa8c34d72c133d97c3f3935b463e5e48"}
Oct 30 22:45:52.715041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:52.714576    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:45:52.835542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:52.835486    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65" podStartSLOduration=1.83547309 podStartE2EDuration="1.83547309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:52.832914345 +0000 UTC m=+3529.167491125" watchObservedRunningTime="2026-10-30 22:45:52.83547309 +0000 UTC m=+3529.170049858"
Oct 30 22:45:56.372401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:45:56.372366    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:45:58.099174 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:45:58.099134    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:46:00.324868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:00.324831    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:46:00.874713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:00.874680    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:03.725115 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:03.725083    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:46:05.427614 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:05.427578    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:05.838835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:05.838758    5503 scope.go:122] "RemoveContainer" containerID="a86ef63bd3878899e6ccc9268e685f53a897d24fa9f2c5c43ab12f90020023fb"
Oct 30 22:46:05.852348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:05.852320    5503 scope.go:122] "RemoveContainer" containerID="f9b392c1dffc23d77f446ab7b821e50a4b27fe0cb2d7da321ce06233d7fe380e"
Oct 30 22:46:10.031068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:10.031034    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:12.114087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:12.114047    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:14.529208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:14.529169    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:19.120698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:19.120658    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:24.045800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:24.045767    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:24.720445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:24.720416    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:25.846712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:25.846677    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:26.148983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:26.148905    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:27.058269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:27.058213    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:27.092405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:27.092365    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"]
Oct 30 22:46:27.137584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:27.137544    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04cfdf32-j6vts"]
Oct 30 22:46:27.269767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:27.269736    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:27.791094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:27.791062    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f03f99d-9ce5-4965-9737-eb50a5d14fbd" path="/var/lib/kubelet/pods/6f03f99d-9ce5-4965-9737-eb50a5d14fbd/volumes"
Oct 30 22:46:28.098778 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:46:28.098700    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:46:28.719618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:28.719581    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:29.924480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:29.924445    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:31.136972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:31.136934    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:31.672681 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:31.672630    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:31.864406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:31.864371    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:32.030866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:32.030791    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:33.560875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:33.560839    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:34.983427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:34.983395    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:35.167512 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:35.167447    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:37.058706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:37.058664    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"]
Oct 30 22:46:37.103033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:37.103001    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5da3c72-s6p4h"]
Oct 30 22:46:37.792109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:37.792064    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716d904d-be7e-4045-b2da-33bb738089cd" path="/var/lib/kubelet/pods/716d904d-be7e-4045-b2da-33bb738089cd/volumes"
Oct 30 22:46:38.423310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:38.423249    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:43.066032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:43.065999    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:46:43.109351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:43.109293    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:44.458987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:44.458919    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:45.556221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:45.556177    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:45.709625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:45.709571    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:45.899047 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:45.898956    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:46.190210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:46.190177    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:46.356827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:46.356794    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:46.623649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:46.623521    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:47.245675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:47.245590    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:47.432810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:47.432774    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:48.074786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:48.074749    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:51.523754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:51.523717    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:52.660456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.660423    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:52.660937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.660757    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"]
Oct 30 22:46:52.665015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.664996    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.721131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.721098    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1b5075\""
Oct 30 22:46:52.721358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.721168    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"agc-cred-f71a8cda136e\""
Oct 30 22:46:52.721358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.721188    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"routing-overlay\""
Oct 30 22:46:52.721358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.721112    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"payload-processing-external-model-e2e--a61972b8-dockercfg-n7gbl\""
Oct 30 22:46:52.772018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.771971    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"kube-root-ca.crt\""
Oct 30 22:46:52.772018 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.771971    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"openshift-service-ca.crt\""
Oct 30 22:46:52.774873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.774850    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"]
Oct 30 22:46:52.802674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.802625    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.802865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.802766    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxv7\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-kube-api-access-pnxv7\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.802865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.802860    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.803002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.802949    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.815124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.815097    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:52.903953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.903915    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.903953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.903962    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.904241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.904036    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.904241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.904094    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxv7\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-kube-api-access-pnxv7\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.904754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.904725    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.904872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.904795    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.906587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.906567    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.966504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.966423    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxv7\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-kube-api-access-pnxv7\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:52.972101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.972074    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:52.978186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:52.978155    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:53.369079 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:46:53.369050    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ceb2e86_e4a3_4c9e_8c29_e863a460f53b.slice/crio-ae3a184ac2231f4eb1f74b80099626ea26bb7b0f58f6797ebbd7f0a47079d316 WatchSource:0}: Error finding container ae3a184ac2231f4eb1f74b80099626ea26bb7b0f58f6797ebbd7f0a47079d316: Status 404 returned error can't find the container with id ae3a184ac2231f4eb1f74b80099626ea26bb7b0f58f6797ebbd7f0a47079d316
Oct 30 22:46:53.371091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:53.371069    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"]
Oct 30 22:46:54.106095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:54.106060    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2" event={"ID":"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b","Type":"ContainerStarted","Data":"3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384"}
Oct 30 22:46:54.106528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:54.106120    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2" event={"ID":"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b","Type":"ContainerStarted","Data":"ae3a184ac2231f4eb1f74b80099626ea26bb7b0f58f6797ebbd7f0a47079d316"}
Oct 30 22:46:54.106528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:54.106287    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:46:54.226833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:54.226782    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2" podStartSLOduration=2.22676718 podStartE2EDuration="2.22676718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:54.224203663 +0000 UTC m=+3590.558780432" watchObservedRunningTime="2026-10-30 22:46:54.22676718 +0000 UTC m=+3590.561343952"
Oct 30 22:46:57.429650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:46:57.429604    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:46:58.098690 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:46:58.098651    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:47:02.042362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:02.042331    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:03.052227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:03.052188    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"]
Oct 30 22:47:03.102596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:03.102566    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f22b95-vsjpp"]
Oct 30 22:47:03.790628 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:03.790597    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aba1e21-48b5-40af-aeac-e19fb071ad27" path="/var/lib/kubelet/pods/8aba1e21-48b5-40af-aeac-e19fb071ad27/volumes"
Oct 30 22:47:04.387551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:04.387523    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:47:04.398952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:04.398930    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:47:05.115764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:05.115736    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:47:06.561376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:06.561342    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:11.108585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:11.108548    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:13.051575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:13.051494    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"]
Oct 30 22:47:13.102097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:13.102067    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2e95ae-f6hvk"]
Oct 30 22:47:13.799119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:13.799085    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd52f46-7694-4088-8ba1-5a2ae3167191" path="/var/lib/kubelet/pods/2dd52f46-7694-4088-8ba1-5a2ae3167191/volumes"
Oct 30 22:47:15.686587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:15.686550    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:20.228820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:20.228766    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:20.401213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:20.401183    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:20.728908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:20.728873    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:25.038050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:25.038022    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:27.285383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.285346    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"]
Oct 30 22:47:27.290773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.290757    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.342884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.342853    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"]
Oct 30 22:47:27.390216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.390175    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:27.390216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.390215    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:47:27.390470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.390300    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:47:27.394950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.394930    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:47:27.440907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.440869    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7j9t\" (UniqueName: \"kubernetes.io/projected/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-kube-api-access-k7j9t\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.441096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.440918    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.441096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.441016    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.542905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.542799    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7j9t\" (UniqueName: \"kubernetes.io/projected/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-kube-api-access-k7j9t\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.542905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.542854    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.543172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.542912    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.543743 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.543689    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.545284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.545263    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.605753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.605714    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7j9t\" (UniqueName: \"kubernetes.io/projected/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-kube-api-access-k7j9t\") pod \"maas-api-revoke-keys-e2e-autores-dcde03-gddxv\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:27.904715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:27.904682    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:28.099467 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:47:28.099428    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:47:28.278456 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:47:28.278415    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5fa03c_0dc2_4f76_957d_75b5fe9d2714.slice/crio-fe854e61ec27114dcfbb806514d00ecae7bd8c99c00d44129f479b5b4121dd21 WatchSource:0}: Error finding container fe854e61ec27114dcfbb806514d00ecae7bd8c99c00d44129f479b5b4121dd21: Status 404 returned error can't find the container with id fe854e61ec27114dcfbb806514d00ecae7bd8c99c00d44129f479b5b4121dd21
Oct 30 22:47:28.280966 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:28.280939    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"]
Oct 30 22:47:28.347048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:28.347013    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv" event={"ID":"af5fa03c-0dc2-4f76-957d-75b5fe9d2714","Type":"ContainerStarted","Data":"fe854e61ec27114dcfbb806514d00ecae7bd8c99c00d44129f479b5b4121dd21"}
Oct 30 22:47:29.355593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:29.355555    5503 generic.go:356] "Generic (PLEG): container finished" podID="af5fa03c-0dc2-4f76-957d-75b5fe9d2714" containerID="73af409bd692502a6aa4b75d60e73addfd203dd1ee63886420aee95d40fc19a4" exitCode=0
Oct 30 22:47:29.356032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:29.355667    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv" event={"ID":"af5fa03c-0dc2-4f76-957d-75b5fe9d2714","Type":"ContainerDied","Data":"73af409bd692502a6aa4b75d60e73addfd203dd1ee63886420aee95d40fc19a4"}
Oct 30 22:47:29.979323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:29.979281    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:30.292142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.292059    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:30.513371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.513343    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:30.582426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.582359    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-kube-api-access-k7j9t\" (UniqueName: \"kubernetes.io/projected/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-kube-api-access-k7j9t\") pod \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") "
Oct 30 22:47:30.582545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.582454    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-openshift-ca\") pod \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") "
Oct 30 22:47:30.582545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.582508    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-xks-ca\") pod \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\" (UID: \"af5fa03c-0dc2-4f76-957d-75b5fe9d2714\") "
Oct 30 22:47:30.582892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.582863    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-openshift-ca" pod "af5fa03c-0dc2-4f76-957d-75b5fe9d2714" (UID: "af5fa03c-0dc2-4f76-957d-75b5fe9d2714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:30.583103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.583084    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:30.584497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.584473    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-xks-ca" pod "af5fa03c-0dc2-4f76-957d-75b5fe9d2714" (UID: "af5fa03c-0dc2-4f76-957d-75b5fe9d2714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:30.584622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.584597    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-kube-api-access-k7j9t" pod "af5fa03c-0dc2-4f76-957d-75b5fe9d2714" (UID: "af5fa03c-0dc2-4f76-957d-75b5fe9d2714"). InnerVolumeSpecName "kube-api-access-k7j9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:30.684296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.684267    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:30.684296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:30.684293    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7j9t\" (UniqueName: \"kubernetes.io/projected/af5fa03c-0dc2-4f76-957d-75b5fe9d2714-kube-api-access-k7j9t\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:31.369523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:31.369491    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"
Oct 30 22:47:31.369721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:31.369526    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv" event={"ID":"af5fa03c-0dc2-4f76-957d-75b5fe9d2714","Type":"ContainerDied","Data":"fe854e61ec27114dcfbb806514d00ecae7bd8c99c00d44129f479b5b4121dd21"}
Oct 30 22:47:31.369721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:31.369572    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe854e61ec27114dcfbb806514d00ecae7bd8c99c00d44129f479b5b4121dd21"
Oct 30 22:47:31.496328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:31.496293    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:34.828983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:34.828949    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:35.461406 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:35.461378    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"]
Oct 30 22:47:35.846025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:35.845935    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:36.021649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:36.021598    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:39.401992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:39.401960    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:40.462715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.462654    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8" podUID="93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" containerName="payload-processing" containerID="cri-o://670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657" gracePeriod=25
Oct 30 22:47:40.730001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.729942    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:47:40.813487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.813459    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-kube-api-access-6dwz5\" (UniqueName: \"kubernetes.io/projected/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-kube-api-access-6dwz5\") pod \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") "
Oct 30 22:47:40.813686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.813651    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-plugins-config-volume\") pod \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\" (UID: \"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c\") "
Oct 30 22:47:40.813995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.813973    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-plugins-config-volume" pod "93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" (UID: "93fa386b-eb7c-48e7-b46e-5e417fe0ed6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:40.815585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.815556    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-kube-api-access-6dwz5" pod "93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" (UID: "93fa386b-eb7c-48e7-b46e-5e417fe0ed6c"). InnerVolumeSpecName "kube-api-access-6dwz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:40.915685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.915629    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwz5\" (UniqueName: \"kubernetes.io/projected/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-kube-api-access-6dwz5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:40.915685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:40.915677    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:41.438068 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.438019    5503 generic.go:356] "Generic (PLEG): container finished" podID="93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" containerID="670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657" exitCode=0
Oct 30 22:47:41.438265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.438083    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8" event={"ID":"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c","Type":"ContainerDied","Data":"670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657"}
Oct 30 22:47:41.438265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.438144    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"
Oct 30 22:47:41.438265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.438210    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8" event={"ID":"93fa386b-eb7c-48e7-b46e-5e417fe0ed6c","Type":"ContainerDied","Data":"a6928f8478570f1f92c7f45e248f625b7af716f8068d2d78ea3ada3891bb73ae"}
Oct 30 22:47:41.438265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.438248    5503 scope.go:122] "RemoveContainer" containerID="670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657"
Oct 30 22:47:41.449472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.449453    5503 scope.go:122] "RemoveContainer" containerID="670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657"
Oct 30 22:47:41.449774 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:47:41.449753    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657\": container with ID starting with 670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657 not found: ID does not exist" containerID="670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657"
Oct 30 22:47:41.449827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.449782    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657"} err="failed to get container status \"670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657\": rpc error: code = NotFound desc = could not find container \"670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657\": container with ID starting with 670583eae9ac60d9835531d17726223a81e1762ce14324f43131953b2108c657 not found: ID does not exist"
Oct 30 22:47:41.609124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.609092    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"]
Oct 30 22:47:41.658242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.658213    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-dcde03-55fb4d67ff-k8rv8"]
Oct 30 22:47:41.789239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:41.789164    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" path="/var/lib/kubelet/pods/93fa386b-eb7c-48e7-b46e-5e417fe0ed6c/volumes"
Oct 30 22:47:43.538435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:43.538402    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:47:44.017951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:44.017925    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:48.587807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:48.587760    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:53.111332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:53.111294    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:57.698118 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:47:57.698083    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:47:58.098548 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:47:58.098470    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:48:02.445931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:02.445893    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:05.938626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:05.938592    5503 scope.go:122] "RemoveContainer" containerID="3ea70ffac6e609df4c7e0b435ceabab8aef0eef9d5c2fc5fcb531069043cba18"
Oct 30 22:48:05.949400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:05.949370    5503 scope.go:122] "RemoveContainer" containerID="8fa4c40f969f2355e8e72562bf13dd2836bdbdd12deb86939a41f99cd54b8c55"
Oct 30 22:48:05.958307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:05.958277    5503 scope.go:122] "RemoveContainer" containerID="2666ad441e4d5b388dff0c968b9313011076ac3f4c5164bd4b1d9b7b11edbaf1"
Oct 30 22:48:05.967600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:05.967582    5503 scope.go:122] "RemoveContainer" containerID="7f09b0906f1369886e43f2153e5a7594bf314adaab246186cb9c3dc7f16a7c28"
Oct 30 22:48:07.022611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:07.022574    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.424531 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:10.424496    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.576350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:10.576314    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.732279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:10.732193    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:10.905579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:10.905536    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.122960 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:11.122883    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.506490 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:11.506455    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.666685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:11.666619    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:11.847243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:11.847161    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:13.585661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:13.585611    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:13.758142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:13.758112    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:16.003309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:16.003270    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:16.160255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:16.160220    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:16.309491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:16.309373    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:19.820594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.820565    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"]
Oct 30 22:48:19.821114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.821098    5503 cpu_manager.go:414] "containerMap: removing container" podUID="93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" containerName="payload-processing"
Oct 30 22:48:19.821114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.821114    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" containerName="payload-processing"
Oct 30 22:48:19.821206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.821129    5503 cpu_manager.go:414] "containerMap: removing container" podUID="af5fa03c-0dc2-4f76-957d-75b5fe9d2714" containerName="revoke-keys"
Oct 30 22:48:19.821206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.821134    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5fa03c-0dc2-4f76-957d-75b5fe9d2714" containerName="revoke-keys"
Oct 30 22:48:19.821324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.821305    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fa386b-eb7c-48e7-b46e-5e417fe0ed6c" containerName="payload-processing"
Oct 30 22:48:19.821373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.821331    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5fa03c-0dc2-4f76-957d-75b5fe9d2714" containerName="revoke-keys"
Oct 30 22:48:19.824781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.824765    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.879802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.879777    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"]
Oct 30 22:48:19.894057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.894019    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44rh\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-kube-api-access-q44rh\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.894234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.894118    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.894309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.894261    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.894537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.894514    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.899305 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.899279    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"agc-cred-d4b0ee4a268a\""
Oct 30 22:48:19.995688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.995628    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.995864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.995807    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.995937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.995916    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44rh\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-kube-api-access-q44rh\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.995998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.995964    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.996450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.996423    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.996559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.996537    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:19.998423 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:19.998402    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:20.057963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.057937    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44rh\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-kube-api-access-q44rh\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:20.138580 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.138517    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:20.221530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.221503    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"]
Oct 30 22:48:20.225776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.225760    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.279537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.279496    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:20.279716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.279582    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:20.279716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.279617    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:20.279813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.279724    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:48:20.281026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.281006    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"]
Oct 30 22:48:20.300077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.300048    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.300249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.300216    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.300331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.300255    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwmgg\" (UniqueName: \"kubernetes.io/projected/4f263e61-3d05-4325-9684-5d2e4e00dbc5-kube-api-access-cwmgg\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.401236 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.401216    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.401377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.401251    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwmgg\" (UniqueName: \"kubernetes.io/projected/4f263e61-3d05-4325-9684-5d2e4e00dbc5-kube-api-access-cwmgg\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.401422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.401373    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.401841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.401817    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.403575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.403554    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.463282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.463249    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwmgg\" (UniqueName: \"kubernetes.io/projected/4f263e61-3d05-4325-9684-5d2e4e00dbc5-kube-api-access-cwmgg\") pod \"maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.509383 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:48:20.509354    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589be902_48e6_468a_bb48_94242a992854.slice/crio-e9307d163488d0f5699e4fb209cd6cea8d26bc7cfc7de0b51e3c31f0369ecdd1 WatchSource:0}: Error finding container e9307d163488d0f5699e4fb209cd6cea8d26bc7cfc7de0b51e3c31f0369ecdd1: Status 404 returned error can't find the container with id e9307d163488d0f5699e4fb209cd6cea8d26bc7cfc7de0b51e3c31f0369ecdd1
Oct 30 22:48:20.511530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.511507    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"]
Oct 30 22:48:20.542184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.542166    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:20.699390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.699305    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt" event={"ID":"589be902-48e6-468a-bb48-94242a992854","Type":"ContainerStarted","Data":"d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78"}
Oct 30 22:48:20.699390 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.699354    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt" event={"ID":"589be902-48e6-468a-bb48-94242a992854","Type":"ContainerStarted","Data":"e9307d163488d0f5699e4fb209cd6cea8d26bc7cfc7de0b51e3c31f0369ecdd1"}
Oct 30 22:48:20.699578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.699501    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:20.820167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.820117    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt" podStartSLOduration=1.8201029690000001 podStartE2EDuration="1.820102969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:20.818170366 +0000 UTC m=+3677.152747118" watchObservedRunningTime="2026-10-30 22:48:20.820102969 +0000 UTC m=+3677.154679771"
Oct 30 22:48:20.920843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:20.920814    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"]
Oct 30 22:48:20.921547 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:48:20.921524    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f263e61_3d05_4325_9684_5d2e4e00dbc5.slice/crio-a532bd52cc50ec9746e210d02a951ff014f2e74eb54eae8cf2c6dfbbd69a8c5c WatchSource:0}: Error finding container a532bd52cc50ec9746e210d02a951ff014f2e74eb54eae8cf2c6dfbbd69a8c5c: Status 404 returned error can't find the container with id a532bd52cc50ec9746e210d02a951ff014f2e74eb54eae8cf2c6dfbbd69a8c5c
Oct 30 22:48:21.087392 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:21.087359    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:21.707480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:21.707440    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc" event={"ID":"4f263e61-3d05-4325-9684-5d2e4e00dbc5","Type":"ContainerStarted","Data":"ed841193b345686ce2be69015c4365c97f191a4d9c1b8e11594eb9f3a15efe92"}
Oct 30 22:48:21.707706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:21.707504    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc" event={"ID":"4f263e61-3d05-4325-9684-5d2e4e00dbc5","Type":"ContainerStarted","Data":"a532bd52cc50ec9746e210d02a951ff014f2e74eb54eae8cf2c6dfbbd69a8c5c"}
Oct 30 22:48:21.832863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:21.832814    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc" podStartSLOduration=1.832799405 podStartE2EDuration="1.832799405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:21.831093279 +0000 UTC m=+3678.165670048" watchObservedRunningTime="2026-10-30 22:48:21.832799405 +0000 UTC m=+3678.167376183"
Oct 30 22:48:22.715479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:22.715445    5503 generic.go:356] "Generic (PLEG): container finished" podID="4f263e61-3d05-4325-9684-5d2e4e00dbc5" containerID="ed841193b345686ce2be69015c4365c97f191a4d9c1b8e11594eb9f3a15efe92" exitCode=0
Oct 30 22:48:22.715902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:22.715522    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc" event={"ID":"4f263e61-3d05-4325-9684-5d2e4e00dbc5","Type":"ContainerDied","Data":"ed841193b345686ce2be69015c4365c97f191a4d9c1b8e11594eb9f3a15efe92"}
Oct 30 22:48:23.873810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:23.873787    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:24.044297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.044218    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-openshift-ca\") pod \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") "
Oct 30 22:48:24.044455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.044339    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f263e61-3d05-4325-9684-5d2e4e00dbc5-kube-api-access-cwmgg\" (UniqueName: \"kubernetes.io/projected/4f263e61-3d05-4325-9684-5d2e4e00dbc5-kube-api-access-cwmgg\") pod \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") "
Oct 30 22:48:24.044455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.044415    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-xks-ca\") pod \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\" (UID: \"4f263e61-3d05-4325-9684-5d2e4e00dbc5\") "
Oct 30 22:48:24.044565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.044538    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-openshift-ca" pod "4f263e61-3d05-4325-9684-5d2e4e00dbc5" (UID: "4f263e61-3d05-4325-9684-5d2e4e00dbc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:24.044771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.044754    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:24.046514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.046487    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-xks-ca" pod "4f263e61-3d05-4325-9684-5d2e4e00dbc5" (UID: "4f263e61-3d05-4325-9684-5d2e4e00dbc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:24.046665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.046578    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f263e61-3d05-4325-9684-5d2e4e00dbc5-kube-api-access-cwmgg" pod "4f263e61-3d05-4325-9684-5d2e4e00dbc5" (UID: "4f263e61-3d05-4325-9684-5d2e4e00dbc5"). InnerVolumeSpecName "kube-api-access-cwmgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:24.145761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.145731    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwmgg\" (UniqueName: \"kubernetes.io/projected/4f263e61-3d05-4325-9684-5d2e4e00dbc5-kube-api-access-cwmgg\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:24.145761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.145758    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f263e61-3d05-4325-9684-5d2e4e00dbc5-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:24.728878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.728851    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"
Oct 30 22:48:24.729067 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.728878    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc" event={"ID":"4f263e61-3d05-4325-9684-5d2e4e00dbc5","Type":"ContainerDied","Data":"a532bd52cc50ec9746e210d02a951ff014f2e74eb54eae8cf2c6dfbbd69a8c5c"}
Oct 30 22:48:24.729067 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:24.728923    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a532bd52cc50ec9746e210d02a951ff014f2e74eb54eae8cf2c6dfbbd69a8c5c"
Oct 30 22:48:25.866160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:25.866126    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:28.099108 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:48:28.099064    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:48:30.052422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:30.052386    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"]
Oct 30 22:48:30.102393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:30.102364    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c0f65-smbgn"]
Oct 30 22:48:30.621140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:30.621112    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:30.773001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:30.772968    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:31.710779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:31.710747    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:48:31.762996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:31.762962    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"]
Oct 30 22:48:31.788864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:31.788839    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ad5a3f-fd20-46b1-8c43-eb9086cd40f3" path="/var/lib/kubelet/pods/44ad5a3f-fd20-46b1-8c43-eb9086cd40f3/volumes"
Oct 30 22:48:31.874444 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:31.874411    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"]
Oct 30 22:48:31.877130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:31.877093    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:31.923003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:31.922973    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"]
Oct 30 22:48:32.148820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:32.148737    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:32.346038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:32.342524    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:35.214232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:35.214200    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:36.763897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:36.763849    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb" podUID="df960647-587c-4ce6-869f-9eecd3c412b9" containerName="payload-processing" containerID="cri-o://dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9" gracePeriod=25
Oct 30 22:48:36.875476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:36.875440    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg" podUID="4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" containerName="payload-pre-processing" containerID="cri-o://0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d" gracePeriod=25
Oct 30 22:48:36.923755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:36.923710    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2" podUID="9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" containerName="payload-processing" containerID="cri-o://3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384" gracePeriod=25
Oct 30 22:48:37.047439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.047412    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:48:37.099570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.099524    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df960647-587c-4ce6-869f-9eecd3c412b9-kube-api-access-k2gk8\" (UniqueName: \"kubernetes.io/projected/df960647-587c-4ce6-869f-9eecd3c412b9-kube-api-access-k2gk8\") pod \"df960647-587c-4ce6-869f-9eecd3c412b9\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") "
Oct 30 22:48:37.099864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.099707    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df960647-587c-4ce6-869f-9eecd3c412b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df960647-587c-4ce6-869f-9eecd3c412b9-plugins-config-volume\") pod \"df960647-587c-4ce6-869f-9eecd3c412b9\" (UID: \"df960647-587c-4ce6-869f-9eecd3c412b9\") "
Oct 30 22:48:37.100386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.100358    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df960647-587c-4ce6-869f-9eecd3c412b9-plugins-config-volume" pod "df960647-587c-4ce6-869f-9eecd3c412b9" (UID: "df960647-587c-4ce6-869f-9eecd3c412b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:37.103131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.103102    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df960647-587c-4ce6-869f-9eecd3c412b9-kube-api-access-k2gk8" pod "df960647-587c-4ce6-869f-9eecd3c412b9" (UID: "df960647-587c-4ce6-869f-9eecd3c412b9"). InnerVolumeSpecName "kube-api-access-k2gk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:37.170475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.170448    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:48:37.200670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.200617    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-plugins-config-volume\") pod \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") "
Oct 30 22:48:37.200878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.200789    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-kube-api-access-l4lnr\" (UniqueName: \"kubernetes.io/projected/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-kube-api-access-l4lnr\") pod \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\" (UID: \"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3\") "
Oct 30 22:48:37.201080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.201057    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-plugins-config-volume" pod "4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" (UID: "4b8ec4e3-61c4-4e4c-a88e-223288db0cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:37.201263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.201241    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.201263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.201260    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2gk8\" (UniqueName: \"kubernetes.io/projected/df960647-587c-4ce6-869f-9eecd3c412b9-kube-api-access-k2gk8\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.201475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.201273    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df960647-587c-4ce6-869f-9eecd3c412b9-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.203111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.203084    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-kube-api-access-l4lnr" pod "4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" (UID: "4b8ec4e3-61c4-4e4c-a88e-223288db0cb3"). InnerVolumeSpecName "kube-api-access-l4lnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:37.216459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.216439    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:48:37.302279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.302191    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-routing-overlay-volume\") pod \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") "
Oct 30 22:48:37.302443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.302279    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-kube-api-access-pnxv7\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-kube-api-access-pnxv7\") pod \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") "
Oct 30 22:48:37.302443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.302309    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-provider-credentials\") pod \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") "
Oct 30 22:48:37.302443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.302417    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-plugins-config-volume\") pod \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\" (UID: \"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b\") "
Oct 30 22:48:37.302674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.302616    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-routing-overlay-volume" pod "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" (UID: "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:37.302937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.302906    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-plugins-config-volume" pod "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" (UID: "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:37.303058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.303043    5503 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-routing-overlay-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.303135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.303065    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4lnr\" (UniqueName: \"kubernetes.io/projected/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3-kube-api-access-l4lnr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.303135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.303081    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.304532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.304511    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-kube-api-access-pnxv7" pod "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" (UID: "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b"). InnerVolumeSpecName "kube-api-access-pnxv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:37.304627 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.304591    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-provider-credentials" pod "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" (UID: "9ceb2e86-e4a3-4c9e-8c29-e863a460f53b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:37.404609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.404576    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnxv7\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-kube-api-access-pnxv7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.404609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.404606    5503 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b-provider-credentials\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:37.814767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.814740    5503 generic.go:356] "Generic (PLEG): container finished" podID="4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" containerID="0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d" exitCode=0
Oct 30 22:48:37.815201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.814792    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"
Oct 30 22:48:37.815201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.814814    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg" event={"ID":"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3","Type":"ContainerDied","Data":"0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d"}
Oct 30 22:48:37.815201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.814869    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg" event={"ID":"4b8ec4e3-61c4-4e4c-a88e-223288db0cb3","Type":"ContainerDied","Data":"96824ff588ccc2c3d9870b2c56ebc6a7c5eb7487bec2ca9b101797e5a0b31b9f"}
Oct 30 22:48:37.815201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.814900    5503 scope.go:122] "RemoveContainer" containerID="0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d"
Oct 30 22:48:37.816753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.816723    5503 generic.go:356] "Generic (PLEG): container finished" podID="9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" containerID="3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384" exitCode=0
Oct 30 22:48:37.817072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.816784    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"
Oct 30 22:48:37.817072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.816891    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2" event={"ID":"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b","Type":"ContainerDied","Data":"3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384"}
Oct 30 22:48:37.817072 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.816939    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2" event={"ID":"9ceb2e86-e4a3-4c9e-8c29-e863a460f53b","Type":"ContainerDied","Data":"ae3a184ac2231f4eb1f74b80099626ea26bb7b0f58f6797ebbd7f0a47079d316"}
Oct 30 22:48:37.818776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.818744    5503 generic.go:356] "Generic (PLEG): container finished" podID="df960647-587c-4ce6-869f-9eecd3c412b9" containerID="dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9" exitCode=0
Oct 30 22:48:37.818886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.818852    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb" event={"ID":"df960647-587c-4ce6-869f-9eecd3c412b9","Type":"ContainerDied","Data":"dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9"}
Oct 30 22:48:37.818940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.818883    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb" event={"ID":"df960647-587c-4ce6-869f-9eecd3c412b9","Type":"ContainerDied","Data":"d08bf88ac208842827fe35c999d0ff8368964df9d80b5cf8db1e76aa00bff501"}
Oct 30 22:48:37.818940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.818914    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"
Oct 30 22:48:37.826262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.826235    5503 scope.go:122] "RemoveContainer" containerID="0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d"
Oct 30 22:48:37.826523 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:48:37.826504    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d\": container with ID starting with 0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d not found: ID does not exist" containerID="0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d"
Oct 30 22:48:37.826572 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.826531    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d"} err="failed to get container status \"0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d\": rpc error: code = NotFound desc = could not find container \"0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d\": container with ID starting with 0a4fd58d596457106a18ffbaeb3af661238b3c3db19ccbdf2e276fb659e5f11d not found: ID does not exist"
Oct 30 22:48:37.826572 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.826547    5503 scope.go:122] "RemoveContainer" containerID="3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384"
Oct 30 22:48:37.834815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.834799    5503 scope.go:122] "RemoveContainer" containerID="3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384"
Oct 30 22:48:37.835083 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:48:37.835061    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384\": container with ID starting with 3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384 not found: ID does not exist" containerID="3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384"
Oct 30 22:48:37.835159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.835093    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384"} err="failed to get container status \"3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384\": rpc error: code = NotFound desc = could not find container \"3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384\": container with ID starting with 3d34353359e4b63da1240e3fc78a2e1760b985e370fecfa2470881ffdc3ba384 not found: ID does not exist"
Oct 30 22:48:37.835159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.835118    5503 scope.go:122] "RemoveContainer" containerID="dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9"
Oct 30 22:48:37.843914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.843890    5503 scope.go:122] "RemoveContainer" containerID="dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9"
Oct 30 22:48:37.844159 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:48:37.844139    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9\": container with ID starting with dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9 not found: ID does not exist" containerID="dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9"
Oct 30 22:48:37.844210 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.844166    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9"} err="failed to get container status \"dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9\": rpc error: code = NotFound desc = could not find container \"dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9\": container with ID starting with dcb26088a4b37ce163c2ff64341364e2e8017956396a84a02d75011870033da9 not found: ID does not exist"
Oct 30 22:48:37.984081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:37.984047    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"]
Oct 30 22:48:38.040331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:38.040251    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b829bc-68fd448855-8plwg"]
Oct 30 22:48:38.104424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:38.104387    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"]
Oct 30 22:48:38.108619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:38.108588    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678x7wq2"]
Oct 30 22:48:38.227056 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:38.227019    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"]
Oct 30 22:48:38.250031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:38.250001    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b829bc-cb6f9fb5f-q7zwb"]
Oct 30 22:48:38.476007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:38.475979    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:48:39.788951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:39.788918    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" path="/var/lib/kubelet/pods/4b8ec4e3-61c4-4e4c-a88e-223288db0cb3/volumes"
Oct 30 22:48:39.789352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:39.789303    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" path="/var/lib/kubelet/pods/9ceb2e86-e4a3-4c9e-8c29-e863a460f53b/volumes"
Oct 30 22:48:39.789670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:39.789654    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df960647-587c-4ce6-869f-9eecd3c412b9" path="/var/lib/kubelet/pods/df960647-587c-4ce6-869f-9eecd3c412b9/volumes"
Oct 30 22:48:39.879728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:39.879694    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:40.052698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:40.052613    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"]
Oct 30 22:48:40.103273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:40.103239    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fef97a57-mqc6n"]
Oct 30 22:48:41.788882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:41.788853    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5093ee1c-dbca-4e6b-8093-8c5087a54ebf" path="/var/lib/kubelet/pods/5093ee1c-dbca-4e6b-8093-8c5087a54ebf/volumes"
Oct 30 22:48:44.813663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:44.813619    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:45.680234 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680199    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"]
Oct 30 22:48:45.680754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680740    5503 cpu_manager.go:414] "containerMap: removing container" podUID="df960647-587c-4ce6-869f-9eecd3c412b9" containerName="payload-processing"
Oct 30 22:48:45.680810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680755    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df960647-587c-4ce6-869f-9eecd3c412b9" containerName="payload-processing"
Oct 30 22:48:45.680810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680794    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" containerName="payload-pre-processing"
Oct 30 22:48:45.680810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680799    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" containerName="payload-pre-processing"
Oct 30 22:48:45.680922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680814    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" containerName="payload-processing"
Oct 30 22:48:45.680922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680820    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" containerName="payload-processing"
Oct 30 22:48:45.680922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680830    5503 cpu_manager.go:414] "containerMap: removing container" podUID="4f263e61-3d05-4325-9684-5d2e4e00dbc5" containerName="revoke-keys"
Oct 30 22:48:45.680922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680834    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f263e61-3d05-4325-9684-5d2e4e00dbc5" containerName="revoke-keys"
Oct 30 22:48:45.680922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680913    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8ec4e3-61c4-4e4c-a88e-223288db0cb3" containerName="payload-pre-processing"
Oct 30 22:48:45.681080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680925    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ceb2e86-e4a3-4c9e-8c29-e863a460f53b" containerName="payload-processing"
Oct 30 22:48:45.681080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680932    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f263e61-3d05-4325-9684-5d2e4e00dbc5" containerName="revoke-keys"
Oct 30 22:48:45.681080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.680938    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="df960647-587c-4ce6-869f-9eecd3c412b9" containerName="payload-processing"
Oct 30 22:48:45.685785 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.685769    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.738188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.738160    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"]
Oct 30 22:48:45.786666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.786614    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:45.786838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.786669    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:48:45.786838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.786614    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:48:45.786838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.786743    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:48:45.809140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.809112    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trj5\" (UniqueName: \"kubernetes.io/projected/a49e4ca4-013b-42ea-8603-cf47b43bc39f-kube-api-access-7trj5\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.809293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.809271    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.809401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.809387    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.911053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.911015    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.911456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.911072    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trj5\" (UniqueName: \"kubernetes.io/projected/a49e4ca4-013b-42ea-8603-cf47b43bc39f-kube-api-access-7trj5\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.911456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.911167    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.911800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.911778    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.913356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.913334    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.973142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.973071    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trj5\" (UniqueName: \"kubernetes.io/projected/a49e4ca4-013b-42ea-8603-cf47b43bc39f-kube-api-access-7trj5\") pod \"maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:45.999194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:45.999168    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:46.369132 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:48:46.369104    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49e4ca4_013b_42ea_8603_cf47b43bc39f.slice/crio-9411be5d92fb59a28d14e01462d989ef7cf91f1d708efb02344e9644e5f5ba30 WatchSource:0}: Error finding container 9411be5d92fb59a28d14e01462d989ef7cf91f1d708efb02344e9644e5f5ba30: Status 404 returned error can't find the container with id 9411be5d92fb59a28d14e01462d989ef7cf91f1d708efb02344e9644e5f5ba30
Oct 30 22:48:46.373401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:46.373371    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"]
Oct 30 22:48:46.889615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:46.889530    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf" event={"ID":"a49e4ca4-013b-42ea-8603-cf47b43bc39f","Type":"ContainerStarted","Data":"8fbddd3a4b98757c8658760dec57b9e2f99c972f1b5f300ab0457d3463840566"}
Oct 30 22:48:46.889615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:46.889583    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf" event={"ID":"a49e4ca4-013b-42ea-8603-cf47b43bc39f","Type":"ContainerStarted","Data":"9411be5d92fb59a28d14e01462d989ef7cf91f1d708efb02344e9644e5f5ba30"}
Oct 30 22:48:47.022541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:47.022479    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf" podStartSLOduration=2.022463737 podStartE2EDuration="2.022463737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:47.020546901 +0000 UTC m=+3703.355123670" watchObservedRunningTime="2026-10-30 22:48:47.022463737 +0000 UTC m=+3703.357040506"
Oct 30 22:48:47.896858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:47.896825    5503 generic.go:356] "Generic (PLEG): container finished" podID="a49e4ca4-013b-42ea-8603-cf47b43bc39f" containerID="8fbddd3a4b98757c8658760dec57b9e2f99c972f1b5f300ab0457d3463840566" exitCode=0
Oct 30 22:48:47.897063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:47.896909    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf" event={"ID":"a49e4ca4-013b-42ea-8603-cf47b43bc39f","Type":"ContainerDied","Data":"8fbddd3a4b98757c8658760dec57b9e2f99c972f1b5f300ab0457d3463840566"}
Oct 30 22:48:49.049125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.049100    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:49.150409 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.150380    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-xks-ca\") pod \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") "
Oct 30 22:48:49.150555 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.150416    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-openshift-ca\") pod \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") "
Oct 30 22:48:49.150555 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.150475    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49e4ca4-013b-42ea-8603-cf47b43bc39f-kube-api-access-7trj5\" (UniqueName: \"kubernetes.io/projected/a49e4ca4-013b-42ea-8603-cf47b43bc39f-kube-api-access-7trj5\") pod \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\" (UID: \"a49e4ca4-013b-42ea-8603-cf47b43bc39f\") "
Oct 30 22:48:49.150798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.150770    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-openshift-ca" pod "a49e4ca4-013b-42ea-8603-cf47b43bc39f" (UID: "a49e4ca4-013b-42ea-8603-cf47b43bc39f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:49.150866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.150846    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:49.152445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.152420    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-xks-ca" pod "a49e4ca4-013b-42ea-8603-cf47b43bc39f" (UID: "a49e4ca4-013b-42ea-8603-cf47b43bc39f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:49.152552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.152504    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49e4ca4-013b-42ea-8603-cf47b43bc39f-kube-api-access-7trj5" pod "a49e4ca4-013b-42ea-8603-cf47b43bc39f" (UID: "a49e4ca4-013b-42ea-8603-cf47b43bc39f"). InnerVolumeSpecName "kube-api-access-7trj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:49.252235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.252207    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49e4ca4-013b-42ea-8603-cf47b43bc39f-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:49.252235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.252233    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trj5\" (UniqueName: \"kubernetes.io/projected/a49e4ca4-013b-42ea-8603-cf47b43bc39f-kube-api-access-7trj5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:49.819350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.819317    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:49.910227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.910190    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"
Oct 30 22:48:49.910227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.910215    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf" event={"ID":"a49e4ca4-013b-42ea-8603-cf47b43bc39f","Type":"ContainerDied","Data":"9411be5d92fb59a28d14e01462d989ef7cf91f1d708efb02344e9644e5f5ba30"}
Oct 30 22:48:49.910455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:49.910276    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9411be5d92fb59a28d14e01462d989ef7cf91f1d708efb02344e9644e5f5ba30"
Oct 30 22:48:53.494943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:53.494907    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"]
Oct 30 22:48:53.495397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:53.495117    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" containerID="cri-o://30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012" gracePeriod=30
Oct 30 22:48:54.528542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:54.528504    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:57.237598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:57.237560    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"]
Oct 30 22:48:57.515845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:57.515765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:57.686592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:57.686551    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:58.098586 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:48:58.098543    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:48:58.673832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.673793    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Oct 30 22:48:58.674331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.673868    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Oct 30 22:48:58.811035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.811006    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:48:58.890193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890100    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-kube-api-access-lqkzr\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-kube-api-access-lqkzr\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890165    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343cb96d-6e76-4dd1-92ad-8794336aa759-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343cb96d-6e76-4dd1-92ad-8794336aa759-istiod-ca-cert\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890193 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890199    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-podinfo\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890264    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-envoy\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890347    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-token\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-token\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890410    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-credential-socket\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890447    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-certs\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890500    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-socket\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.890886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.890630    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-data\") pod \"343cb96d-6e76-4dd1-92ad-8794336aa759\" (UID: \"343cb96d-6e76-4dd1-92ad-8794336aa759\") "
Oct 30 22:48:58.892036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.891690    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-data" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:58.892391 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.892361    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-certs" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:58.892759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.892551    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-credential-socket" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:58.893273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.893025    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343cb96d-6e76-4dd1-92ad-8794336aa759-istiod-ca-cert" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:58.895666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.895613    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-socket" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:58.896263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.896239    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-envoy" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:58.896769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.896735    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-podinfo" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:58.896992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.896964    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-kube-api-access-lqkzr" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "kube-api-access-lqkzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:58.897162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.897136    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-token" pod "343cb96d-6e76-4dd1-92ad-8794336aa759" (UID: "343cb96d-6e76-4dd1-92ad-8794336aa759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:58.968469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.968426    5503 generic.go:356] "Generic (PLEG): container finished" podID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerID="30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012" exitCode=0
Oct 30 22:48:58.968703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.968493    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"
Oct 30 22:48:58.968703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.968520    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" event={"ID":"343cb96d-6e76-4dd1-92ad-8794336aa759","Type":"ContainerDied","Data":"30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012"}
Oct 30 22:48:58.968703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.968595    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks" event={"ID":"343cb96d-6e76-4dd1-92ad-8794336aa759","Type":"ContainerDied","Data":"64970a52d5bb734e85c306bb3d7fa5f2c2841a7b344a77b6cdb3457f24573ae0"}
Oct 30 22:48:58.968703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.968630    5503 scope.go:122] "RemoveContainer" containerID="30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012"
Oct 30 22:48:58.979929 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.979912    5503 scope.go:122] "RemoveContainer" containerID="30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012"
Oct 30 22:48:58.980176 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:48:58.980158    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012\": container with ID starting with 30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012 not found: ID does not exist" containerID="30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012"
Oct 30 22:48:58.980216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.980186    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012"} err="failed to get container status \"30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012\": rpc error: code = NotFound desc = could not find container \"30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012\": container with ID starting with 30d5342013db1035c8d7a183cd6a8047cbeff0ae7c5d5b4f0cd3113c4e9dd012 not found: ID does not exist"
Oct 30 22:48:58.992387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992364    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992389    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992399    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqkzr\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-kube-api-access-lqkzr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992409    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/343cb96d-6e76-4dd1-92ad-8794336aa759-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992418    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992428    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992436    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/343cb96d-6e76-4dd1-92ad-8794336aa759-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992444    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.992476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:58.992452    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/343cb96d-6e76-4dd1-92ad-8794336aa759-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:59.156752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:59.156695    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:48:59.157680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:59.157589    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"]
Oct 30 22:48:59.161398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:59.161367    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-7bvks"]
Oct 30 22:48:59.789618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:48:59.789580    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" path="/var/lib/kubelet/pods/343cb96d-6e76-4dd1-92ad-8794336aa759/volumes"
Oct 30 22:49:02.238123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.238083    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65" podUID="71c991ba-8682-46ce-8b06-10af066a8e4e" containerName="payload-pre-processing" containerID="cri-o://a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600" gracePeriod=25
Oct 30 22:49:02.499057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.498994    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:49:02.532449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.532417    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c991ba-8682-46ce-8b06-10af066a8e4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c991ba-8682-46ce-8b06-10af066a8e4e-plugins-config-volume\") pod \"71c991ba-8682-46ce-8b06-10af066a8e4e\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") "
Oct 30 22:49:02.532663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.532535    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c991ba-8682-46ce-8b06-10af066a8e4e-kube-api-access-h6b77\" (UniqueName: \"kubernetes.io/projected/71c991ba-8682-46ce-8b06-10af066a8e4e-kube-api-access-h6b77\") pod \"71c991ba-8682-46ce-8b06-10af066a8e4e\" (UID: \"71c991ba-8682-46ce-8b06-10af066a8e4e\") "
Oct 30 22:49:02.532902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.532866    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c991ba-8682-46ce-8b06-10af066a8e4e-plugins-config-volume" pod "71c991ba-8682-46ce-8b06-10af066a8e4e" (UID: "71c991ba-8682-46ce-8b06-10af066a8e4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:02.533141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.533113    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c991ba-8682-46ce-8b06-10af066a8e4e-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:02.534946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.534918    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c991ba-8682-46ce-8b06-10af066a8e4e-kube-api-access-h6b77" pod "71c991ba-8682-46ce-8b06-10af066a8e4e" (UID: "71c991ba-8682-46ce-8b06-10af066a8e4e"). InnerVolumeSpecName "kube-api-access-h6b77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:02.634415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.634383    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6b77\" (UniqueName: \"kubernetes.io/projected/71c991ba-8682-46ce-8b06-10af066a8e4e-kube-api-access-h6b77\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:02.995173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.995137    5503 generic.go:356] "Generic (PLEG): container finished" podID="71c991ba-8682-46ce-8b06-10af066a8e4e" containerID="a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600" exitCode=0
Oct 30 22:49:02.995364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.995216    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"
Oct 30 22:49:02.995364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.995226    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65" event={"ID":"71c991ba-8682-46ce-8b06-10af066a8e4e","Type":"ContainerDied","Data":"a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600"}
Oct 30 22:49:02.995364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.995285    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65" event={"ID":"71c991ba-8682-46ce-8b06-10af066a8e4e","Type":"ContainerDied","Data":"ba27eb6f3183fa0aec72cc2017fa6f27aa8c34d72c133d97c3f3935b463e5e48"}
Oct 30 22:49:02.995364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:02.995320    5503 scope.go:122] "RemoveContainer" containerID="a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600"
Oct 30 22:49:03.005698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:03.005615    5503 scope.go:122] "RemoveContainer" containerID="a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600"
Oct 30 22:49:03.006008 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:49:03.005976    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600\": container with ID starting with a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600 not found: ID does not exist" containerID="a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600"
Oct 30 22:49:03.006128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:03.006018    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600"} err="failed to get container status \"a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600\": rpc error: code = NotFound desc = could not find container \"a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600\": container with ID starting with a731521ffed066cfac6ce281e9ecf37ff29e635b791d5d211dabd5220195f600 not found: ID does not exist"
Oct 30 22:49:03.086111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:03.086075    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:03.165204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:03.165164    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"]
Oct 30 22:49:03.168670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:03.168619    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-031eee-59b67ccc55-9sw65"]
Oct 30 22:49:03.789114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:03.789085    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c991ba-8682-46ce-8b06-10af066a8e4e" path="/var/lib/kubelet/pods/71c991ba-8682-46ce-8b06-10af066a8e4e/volumes"
Oct 30 22:49:04.517172 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:04.517131    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:04.735877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:04.735841    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:04.895106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:04.895009    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:05.175316 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:05.175284    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"]
Oct 30 22:49:05.225858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:05.225824    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-031eee-gmhkf"]
Oct 30 22:49:05.584375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:05.584280    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:05.788649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:05.788613    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49e4ca4-013b-42ea-8603-cf47b43bc39f" path="/var/lib/kubelet/pods/a49e4ca4-013b-42ea-8603-cf47b43bc39f/volumes"
Oct 30 22:49:06.007348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:06.007318    5503 scope.go:122] "RemoveContainer" containerID="d28fadb9a686d6dae223914eacc60c42287ea2a2c0e7ede3cf371e4b11b80d77"
Oct 30 22:49:09.031796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:09.031741    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:10.643033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:10.643005    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:10.914933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:10.914900    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:11.077502 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:11.077436    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:11.165626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:11.165553    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"]
Oct 30 22:49:11.216114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:11.216086    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-dcde03-gddxv"]
Oct 30 22:49:11.491958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:11.491876    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:11.647596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:11.647539    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:11.789661 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:11.789564    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5fa03c-0dc2-4f76-957d-75b5fe9d2714" path="/var/lib/kubelet/pods/af5fa03c-0dc2-4f76-957d-75b5fe9d2714/volumes"
Oct 30 22:49:12.559144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:12.559106    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:12.732098 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:12.732046    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:12.943382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:12.943350    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:14.996682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:14.996627    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:15.531944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.531914    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"]
Oct 30 22:49:15.532399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532386    5503 cpu_manager.go:414] "containerMap: removing container" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy"
Oct 30 22:49:15.532463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532400    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy"
Oct 30 22:49:15.532463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532418    5503 cpu_manager.go:414] "containerMap: removing container" podUID="71c991ba-8682-46ce-8b06-10af066a8e4e" containerName="payload-pre-processing"
Oct 30 22:49:15.532463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532424    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c991ba-8682-46ce-8b06-10af066a8e4e" containerName="payload-pre-processing"
Oct 30 22:49:15.532463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532460    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a49e4ca4-013b-42ea-8603-cf47b43bc39f" containerName="revoke-keys"
Oct 30 22:49:15.532463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532464    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49e4ca4-013b-42ea-8603-cf47b43bc39f" containerName="revoke-keys"
Oct 30 22:49:15.532660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532548    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49e4ca4-013b-42ea-8603-cf47b43bc39f" containerName="revoke-keys"
Oct 30 22:49:15.532660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532558    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="343cb96d-6e76-4dd1-92ad-8794336aa759" containerName="istio-proxy"
Oct 30 22:49:15.532660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.532567    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c991ba-8682-46ce-8b06-10af066a8e4e" containerName="payload-pre-processing"
Oct 30 22:49:15.535945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.535930    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.590688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.590661    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"]
Oct 30 22:49:15.638418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.638387    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:15.638418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.638393    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:49:15.638677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.638424    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:15.638677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.638464    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:15.698820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.698786    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.699014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.698828    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dngqt\" (UniqueName: \"kubernetes.io/projected/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-kube-api-access-dngqt\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.699014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.698900    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.799707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.799619    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.799859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.799746    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.799859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.799780    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dngqt\" (UniqueName: \"kubernetes.io/projected/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-kube-api-access-dngqt\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.800439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.800418    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.801952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.801929    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:15.819619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.819594    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:15.866322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:15.866283    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dngqt\" (UniqueName: \"kubernetes.io/projected/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-kube-api-access-dngqt\") pod \"maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:16.149937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.149854    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:16.521740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.521709    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"]
Oct 30 22:49:16.529651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.529616    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.572040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.572016    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"]
Oct 30 22:49:16.579712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.579688    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"]
Oct 30 22:49:16.584481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.584463    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1b5075\"/\"agc-cred-12328dd500c5\""
Oct 30 22:49:16.710094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.710053    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdqq5\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-kube-api-access-wdqq5\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.710281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.710116    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.710281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.710244    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.710387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.710343    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.811165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.811087    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.811309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.811222    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdqq5\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-kube-api-access-wdqq5\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.811309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.811271    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.811395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.811312    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.811920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.811892    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.812039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.812017    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.813954 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.813931    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:16.908768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:16.908732    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdqq5\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-kube-api-access-wdqq5\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:17.104089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:17.104008    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp" event={"ID":"7f5c6c8f-f522-41a0-8970-5166b4d6bb02","Type":"ContainerStarted","Data":"b491c817225a57a9ac270aec660254bc6858c7fcdc5db93441cc96c1d334398b"}
Oct 30 22:49:17.104089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:17.104058    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp" event={"ID":"7f5c6c8f-f522-41a0-8970-5166b4d6bb02","Type":"ContainerStarted","Data":"39b242628a047e49257fa79e180ef575d525d8cc0e14eec64d8fa5701ccd3199"}
Oct 30 22:49:17.152732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:17.152696    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:17.229011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:17.228946    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp" podStartSLOduration=2.228927497 podStartE2EDuration="2.228927497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:17.226013585 +0000 UTC m=+3733.560590348" watchObservedRunningTime="2026-10-30 22:49:17.228927497 +0000 UTC m=+3733.563504267"
Oct 30 22:49:17.529494 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:49:17.529462    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fc80fd_a0d8_49c4_a724_83ae164b54d9.slice/crio-c560b5d0c62dcf58ccea0bf5e04cc4df31c47683389833378359968fcaf5efb8 WatchSource:0}: Error finding container c560b5d0c62dcf58ccea0bf5e04cc4df31c47683389833378359968fcaf5efb8: Status 404 returned error can't find the container with id c560b5d0c62dcf58ccea0bf5e04cc4df31c47683389833378359968fcaf5efb8
Oct 30 22:49:17.532738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:17.532701    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"]
Oct 30 22:49:18.111654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.111554    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" event={"ID":"b0fc80fd-a0d8-49c4-a724-83ae164b54d9","Type":"ContainerStarted","Data":"49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe"}
Oct 30 22:49:18.111654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.111606    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" event={"ID":"b0fc80fd-a0d8-49c4-a724-83ae164b54d9","Type":"ContainerStarted","Data":"c560b5d0c62dcf58ccea0bf5e04cc4df31c47683389833378359968fcaf5efb8"}
Oct 30 22:49:18.111901 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.111769    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:18.113277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.113254    5503 generic.go:356] "Generic (PLEG): container finished" podID="7f5c6c8f-f522-41a0-8970-5166b4d6bb02" containerID="b491c817225a57a9ac270aec660254bc6858c7fcdc5db93441cc96c1d334398b" exitCode=0
Oct 30 22:49:18.113393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.113308    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp" event={"ID":"7f5c6c8f-f522-41a0-8970-5166b4d6bb02","Type":"ContainerDied","Data":"b491c817225a57a9ac270aec660254bc6858c7fcdc5db93441cc96c1d334398b"}
Oct 30 22:49:18.232983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.232930    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" podStartSLOduration=2.23291651 podStartE2EDuration="2.23291651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:18.232199892 +0000 UTC m=+3734.566776662" watchObservedRunningTime="2026-10-30 22:49:18.23291651 +0000 UTC m=+3734.567493279"
Oct 30 22:49:18.242522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.242494    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:18.409154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.409121    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:18.597451 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:18.597418    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:19.267129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.267102    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:19.444218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.444190    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-kube-api-access-dngqt\" (UniqueName: \"kubernetes.io/projected/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-kube-api-access-dngqt\") pod \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") "
Oct 30 22:49:19.444375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.444327    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-xks-ca\") pod \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") "
Oct 30 22:49:19.444484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.444387    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-openshift-ca\") pod \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\" (UID: \"7f5c6c8f-f522-41a0-8970-5166b4d6bb02\") "
Oct 30 22:49:19.444814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.444785    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-openshift-ca" pod "7f5c6c8f-f522-41a0-8970-5166b4d6bb02" (UID: "7f5c6c8f-f522-41a0-8970-5166b4d6bb02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:19.446281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.446254    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-kube-api-access-dngqt" pod "7f5c6c8f-f522-41a0-8970-5166b4d6bb02" (UID: "7f5c6c8f-f522-41a0-8970-5166b4d6bb02"). InnerVolumeSpecName "kube-api-access-dngqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:19.446383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.446314    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-xks-ca" pod "7f5c6c8f-f522-41a0-8970-5166b4d6bb02" (UID: "7f5c6c8f-f522-41a0-8970-5166b4d6bb02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:19.546006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.545977    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dngqt\" (UniqueName: \"kubernetes.io/projected/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-kube-api-access-dngqt\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:19.546006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.546003    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:19.546175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.546014    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c6c8f-f522-41a0-8970-5166b4d6bb02-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:19.973846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:19.973809    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"]
Oct 30 22:49:20.124855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:20.124728    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:20.125560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:20.125540    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"
Oct 30 22:49:20.125702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:20.125567    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp" event={"ID":"7f5c6c8f-f522-41a0-8970-5166b4d6bb02","Type":"ContainerDied","Data":"39b242628a047e49257fa79e180ef575d525d8cc0e14eec64d8fa5701ccd3199"}
Oct 30 22:49:20.125702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:20.125608    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39b242628a047e49257fa79e180ef575d525d8cc0e14eec64d8fa5701ccd3199"
Oct 30 22:49:20.670776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:20.670747    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:20.883054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:20.883017    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:21.421113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:21.421080    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:21.727590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:21.727494    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:25.856656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:25.856603    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:26.384577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:26.384540    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:26.549755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:26.549721    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:27.112867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:27.112836    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:49:27.632110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:27.632075    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:27.794425 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:27.794395    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:28.099742 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:49:28.099707    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:49:28.243402 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:28.243368    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:28.409550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:28.409515    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:29.125992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:29.125934    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:49:29.299685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:29.299624    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"]
Oct 30 22:49:30.727728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:30.727691    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:30.966541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:30.966510    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.304199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.304168    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.483317 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.483288    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:31.633292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.633203    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"]
Oct 30 22:49:31.634557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.634525    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7f5c6c8f-f522-41a0-8970-5166b4d6bb02" containerName="revoke-keys"
Oct 30 22:49:31.634786 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.634767    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5c6c8f-f522-41a0-8970-5166b4d6bb02" containerName="revoke-keys"
Oct 30 22:49:31.635055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.635042    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5c6c8f-f522-41a0-8970-5166b4d6bb02" containerName="revoke-keys"
Oct 30 22:49:31.639625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.639603    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.677289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.677255    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"]
Oct 30 22:49:31.691145 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.691113    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.691318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.691190    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpqd\" (UniqueName: \"kubernetes.io/projected/e890ee22-58d5-495d-9cc3-ccf8fb863093-kube-api-access-dtpqd\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.691318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.691261    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.694114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.694084    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:31.725769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.725734    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:49:31.725769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.725765    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:49:31.725974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.725770    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:49:31.747322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.747295    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"]
Oct 30 22:49:31.751314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.751296    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:31.791890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.791866    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"]
Oct 30 22:49:31.792415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.792398    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.792481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.792428    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9ml\" (UniqueName: \"kubernetes.io/projected/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-kube-api-access-hq9ml\") pod \"payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:31.792547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.792532    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:31.792740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.792717    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.792791 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.792756    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpqd\" (UniqueName: \"kubernetes.io/projected/e890ee22-58d5-495d-9cc3-ccf8fb863093-kube-api-access-dtpqd\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.793245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.793219    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.795093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.795071    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.815014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.814988    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9626a6\""
Oct 30 22:49:31.815296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.815277    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9626a6-dockercfg-2bn2q\""
Oct 30 22:49:31.873244 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.872952    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"]
Oct 30 22:49:31.899703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.880411    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpqd\" (UniqueName: \"kubernetes.io/projected/e890ee22-58d5-495d-9cc3-ccf8fb863093-kube-api-access-dtpqd\") pod \"maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.899703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.884239    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:31.899703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.894393    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:31.899703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.894562    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9ml\" (UniqueName: \"kubernetes.io/projected/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-kube-api-access-hq9ml\") pod \"payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:31.899703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.895306    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:31.927550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.927520    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"]
Oct 30 22:49:31.955927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.955894    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:31.960463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.960439    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9ml\" (UniqueName: \"kubernetes.io/projected/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-kube-api-access-hq9ml\") pod \"payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:31.995386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.995353    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d20a62-5c5e-4de8-b950-67819409cbc7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:31.995534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:31.995434    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdwq\" (UniqueName: \"kubernetes.io/projected/06d20a62-5c5e-4de8-b950-67819409cbc7-kube-api-access-6hdwq\") pod \"payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:32.065464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.065431    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:32.096415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.096389    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdwq\" (UniqueName: \"kubernetes.io/projected/06d20a62-5c5e-4de8-b950-67819409cbc7-kube-api-access-6hdwq\") pod \"payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:32.096591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.096572    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d20a62-5c5e-4de8-b950-67819409cbc7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:32.097257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.097234    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d20a62-5c5e-4de8-b950-67819409cbc7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:32.158812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.158785    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdwq\" (UniqueName: \"kubernetes.io/projected/06d20a62-5c5e-4de8-b950-67819409cbc7-kube-api-access-6hdwq\") pod \"payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:32.211432 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.211400    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:32.221403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.221376    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:32.333915 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:49:32.333775    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode890ee22_58d5_495d_9cc3_ccf8fb863093.slice/crio-0ae77bcc86d22f4f33f36d68fa38c349cfc3e8ca0e0e4d9bc762852174092701 WatchSource:0}: Error finding container 0ae77bcc86d22f4f33f36d68fa38c349cfc3e8ca0e0e4d9bc762852174092701: Status 404 returned error can't find the container with id 0ae77bcc86d22f4f33f36d68fa38c349cfc3e8ca0e0e4d9bc762852174092701
Oct 30 22:49:32.336866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.336838    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"]
Oct 30 22:49:32.441873 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:49:32.441839    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed8f02a_3a4e_4ee9_9c60_d7b7445a8981.slice/crio-6ff6794366acc46108a2790de97b5ccb2d10ba0a6228eff5ea6d8d16e8b229b8 WatchSource:0}: Error finding container 6ff6794366acc46108a2790de97b5ccb2d10ba0a6228eff5ea6d8d16e8b229b8: Status 404 returned error can't find the container with id 6ff6794366acc46108a2790de97b5ccb2d10ba0a6228eff5ea6d8d16e8b229b8
Oct 30 22:49:32.444078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.444058    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"]
Oct 30 22:49:32.583484 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:49:32.583452    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d20a62_5c5e_4de8_b950_67819409cbc7.slice/crio-96b45cfe6a3decbeec61c7ecf8bdb269549acbc9b307aafdb72c38bd9503097b WatchSource:0}: Error finding container 96b45cfe6a3decbeec61c7ecf8bdb269549acbc9b307aafdb72c38bd9503097b: Status 404 returned error can't find the container with id 96b45cfe6a3decbeec61c7ecf8bdb269549acbc9b307aafdb72c38bd9503097b
Oct 30 22:49:32.586472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:32.586444    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"]
Oct 30 22:49:33.209523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.209483    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2" event={"ID":"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981","Type":"ContainerStarted","Data":"cb8cd522697bb61196d180dafe90871c840e3749f652d36f047ca04cd5714c85"}
Oct 30 22:49:33.210025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.209555    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2" event={"ID":"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981","Type":"ContainerStarted","Data":"6ff6794366acc46108a2790de97b5ccb2d10ba0a6228eff5ea6d8d16e8b229b8"}
Oct 30 22:49:33.210025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.209673    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:33.211190 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.211160    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw" event={"ID":"e890ee22-58d5-495d-9cc3-ccf8fb863093","Type":"ContainerStarted","Data":"1c76427e8a76d5cc927603fb3defaf75a85c6973f7c68af898b97c242c9aae01"}
Oct 30 22:49:33.211330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.211212    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw" event={"ID":"e890ee22-58d5-495d-9cc3-ccf8fb863093","Type":"ContainerStarted","Data":"0ae77bcc86d22f4f33f36d68fa38c349cfc3e8ca0e0e4d9bc762852174092701"}
Oct 30 22:49:33.212942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.212918    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb" event={"ID":"06d20a62-5c5e-4de8-b950-67819409cbc7","Type":"ContainerStarted","Data":"64c18d1dd969388dd6caada9a43ba6a49a08976e9d92e1415a70f3689a0e5bf3"}
Oct 30 22:49:33.213062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.212955    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb" event={"ID":"06d20a62-5c5e-4de8-b950-67819409cbc7","Type":"ContainerStarted","Data":"96b45cfe6a3decbeec61c7ecf8bdb269549acbc9b307aafdb72c38bd9503097b"}
Oct 30 22:49:33.213062 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.212999    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:33.328657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.328598    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2" podStartSLOduration=2.328584017 podStartE2EDuration="2.328584017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:33.326874272 +0000 UTC m=+3749.661451042" watchObservedRunningTime="2026-10-30 22:49:33.328584017 +0000 UTC m=+3749.663160786"
Oct 30 22:49:33.449102 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.448077    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw" podStartSLOduration=2.448056843 podStartE2EDuration="2.448056843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:33.446419892 +0000 UTC m=+3749.780996663" watchObservedRunningTime="2026-10-30 22:49:33.448056843 +0000 UTC m=+3749.782633613"
Oct 30 22:49:33.567529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:33.567433    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb" podStartSLOduration=2.567420387 podStartE2EDuration="2.567420387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:33.564914275 +0000 UTC m=+3749.899491044" watchObservedRunningTime="2026-10-30 22:49:33.567420387 +0000 UTC m=+3749.901997156"
Oct 30 22:49:34.220977 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.220941    5503 generic.go:356] "Generic (PLEG): container finished" podID="e890ee22-58d5-495d-9cc3-ccf8fb863093" containerID="1c76427e8a76d5cc927603fb3defaf75a85c6973f7c68af898b97c242c9aae01" exitCode=0
Oct 30 22:49:34.221348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.221028    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw" event={"ID":"e890ee22-58d5-495d-9cc3-ccf8fb863093","Type":"ContainerDied","Data":"1c76427e8a76d5cc927603fb3defaf75a85c6973f7c68af898b97c242c9aae01"}
Oct 30 22:49:34.300465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.300417    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt" podUID="589be902-48e6-468a-bb48-94242a992854" containerName="payload-processing" containerID="cri-o://d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78" gracePeriod=25
Oct 30 22:49:34.560226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.560203    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:49:34.625938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.625915    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-routing-overlay-volume\") pod \"589be902-48e6-468a-bb48-94242a992854\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") "
Oct 30 22:49:34.626085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.625948    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-kube-api-access-q44rh\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-kube-api-access-q44rh\") pod \"589be902-48e6-468a-bb48-94242a992854\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") "
Oct 30 22:49:34.626085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.625978    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-provider-credentials\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-provider-credentials\") pod \"589be902-48e6-468a-bb48-94242a992854\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") "
Oct 30 22:49:34.626085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.626072    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-plugins-config-volume\") pod \"589be902-48e6-468a-bb48-94242a992854\" (UID: \"589be902-48e6-468a-bb48-94242a992854\") "
Oct 30 22:49:34.626458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.626435    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-plugins-config-volume" pod "589be902-48e6-468a-bb48-94242a992854" (UID: "589be902-48e6-468a-bb48-94242a992854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:34.626562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.626493    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-routing-overlay-volume" pod "589be902-48e6-468a-bb48-94242a992854" (UID: "589be902-48e6-468a-bb48-94242a992854"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:34.626616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.626568    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:34.628135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.628114    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-provider-credentials" pod "589be902-48e6-468a-bb48-94242a992854" (UID: "589be902-48e6-468a-bb48-94242a992854"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:34.628260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.628145    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-kube-api-access-q44rh" pod "589be902-48e6-468a-bb48-94242a992854" (UID: "589be902-48e6-468a-bb48-94242a992854"). InnerVolumeSpecName "kube-api-access-q44rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:34.727480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.727447    5503 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/589be902-48e6-468a-bb48-94242a992854-routing-overlay-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:34.727480 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.727481    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44rh\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-kube-api-access-q44rh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:34.727704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.727490    5503 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/589be902-48e6-468a-bb48-94242a992854-provider-credentials\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:34.994514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:34.994471    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main" containerID="cri-o://e2efe4261aa23125a83c8fffdefdf882080a60ac16c72beab0ec864a8179fa37" gracePeriod=45
Oct 30 22:49:35.229899 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.229860    5503 generic.go:356] "Generic (PLEG): container finished" podID="589be902-48e6-468a-bb48-94242a992854" containerID="d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78" exitCode=0
Oct 30 22:49:35.230355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.229922    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"
Oct 30 22:49:35.230355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.229920    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt" event={"ID":"589be902-48e6-468a-bb48-94242a992854","Type":"ContainerDied","Data":"d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78"}
Oct 30 22:49:35.230355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.230059    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt" event={"ID":"589be902-48e6-468a-bb48-94242a992854","Type":"ContainerDied","Data":"e9307d163488d0f5699e4fb209cd6cea8d26bc7cfc7de0b51e3c31f0369ecdd1"}
Oct 30 22:49:35.230355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.230092    5503 scope.go:122] "RemoveContainer" containerID="d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78"
Oct 30 22:49:35.232594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.232493    5503 generic.go:356] "Generic (PLEG): container finished" podID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerID="e2efe4261aa23125a83c8fffdefdf882080a60ac16c72beab0ec864a8179fa37" exitCode=0
Oct 30 22:49:35.232594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.232542    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" event={"ID":"eea4a889-5455-4f4a-b278-31c0d2d45d84","Type":"ContainerDied","Data":"e2efe4261aa23125a83c8fffdefdf882080a60ac16c72beab0ec864a8179fa37"}
Oct 30 22:49:35.241101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.241079    5503 scope.go:122] "RemoveContainer" containerID="d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78"
Oct 30 22:49:35.241589 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:49:35.241435    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78\": container with ID starting with d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78 not found: ID does not exist" containerID="d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78"
Oct 30 22:49:35.241589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.241487    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78"} err="failed to get container status \"d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78\": rpc error: code = NotFound desc = could not find container \"d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78\": container with ID starting with d7d84f896716ee2b1ce60f509c398c2896bcf9d4620924e4f3c2657ce5299f78 not found: ID does not exist"
Oct 30 22:49:35.246860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.246808    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:49:35.333759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.333734    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-model-cache\") pod \"eea4a889-5455-4f4a-b278-31c0d2d45d84\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") "
Oct 30 22:49:35.333907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.333832    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eea4a889-5455-4f4a-b278-31c0d2d45d84-tls-certs\" (UniqueName: \"kubernetes.io/secret/eea4a889-5455-4f4a-b278-31c0d2d45d84-tls-certs\") pod \"eea4a889-5455-4f4a-b278-31c0d2d45d84\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") "
Oct 30 22:49:35.333907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.333853    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-tmp-dir\") pod \"eea4a889-5455-4f4a-b278-31c0d2d45d84\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") "
Oct 30 22:49:35.333907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.333874    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-home\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-home\") pod \"eea4a889-5455-4f4a-b278-31c0d2d45d84\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") "
Oct 30 22:49:35.334094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.333990    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-model-cache" pod "eea4a889-5455-4f4a-b278-31c0d2d45d84" (UID: "eea4a889-5455-4f4a-b278-31c0d2d45d84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:35.334094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.334018    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea4a889-5455-4f4a-b278-31c0d2d45d84-kube-api-access-gbltk\" (UniqueName: \"kubernetes.io/projected/eea4a889-5455-4f4a-b278-31c0d2d45d84-kube-api-access-gbltk\") pod \"eea4a889-5455-4f4a-b278-31c0d2d45d84\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") "
Oct 30 22:49:35.334206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.334101    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-dshm\") pod \"eea4a889-5455-4f4a-b278-31c0d2d45d84\" (UID: \"eea4a889-5455-4f4a-b278-31c0d2d45d84\") "
Oct 30 22:49:35.334206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.334158    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-tmp-dir" pod "eea4a889-5455-4f4a-b278-31c0d2d45d84" (UID: "eea4a889-5455-4f4a-b278-31c0d2d45d84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:35.334206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.334167    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-home" pod "eea4a889-5455-4f4a-b278-31c0d2d45d84" (UID: "eea4a889-5455-4f4a-b278-31c0d2d45d84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:35.334562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.334540    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.334686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.334567    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.334686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.334581    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.336679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.336650    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea4a889-5455-4f4a-b278-31c0d2d45d84-tls-certs" pod "eea4a889-5455-4f4a-b278-31c0d2d45d84" (UID: "eea4a889-5455-4f4a-b278-31c0d2d45d84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:35.336802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.336760    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea4a889-5455-4f4a-b278-31c0d2d45d84-kube-api-access-gbltk" pod "eea4a889-5455-4f4a-b278-31c0d2d45d84" (UID: "eea4a889-5455-4f4a-b278-31c0d2d45d84"). InnerVolumeSpecName "kube-api-access-gbltk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:35.337779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.337752    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-dshm" pod "eea4a889-5455-4f4a-b278-31c0d2d45d84" (UID: "eea4a889-5455-4f4a-b278-31c0d2d45d84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:35.392303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.392276    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:35.436671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.435189    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e890ee22-58d5-495d-9cc3-ccf8fb863093-kube-api-access-dtpqd\" (UniqueName: \"kubernetes.io/projected/e890ee22-58d5-495d-9cc3-ccf8fb863093-kube-api-access-dtpqd\") pod \"e890ee22-58d5-495d-9cc3-ccf8fb863093\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") "
Oct 30 22:49:35.436671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.435259    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-xks-ca\") pod \"e890ee22-58d5-495d-9cc3-ccf8fb863093\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") "
Oct 30 22:49:35.436671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.435403    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-openshift-ca\") pod \"e890ee22-58d5-495d-9cc3-ccf8fb863093\" (UID: \"e890ee22-58d5-495d-9cc3-ccf8fb863093\") "
Oct 30 22:49:35.436671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.435871    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea4a889-5455-4f4a-b278-31c0d2d45d84-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.436671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.435897    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea4a889-5455-4f4a-b278-31c0d2d45d84-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.436671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.435909    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbltk\" (UniqueName: \"kubernetes.io/projected/eea4a889-5455-4f4a-b278-31c0d2d45d84-kube-api-access-gbltk\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.436671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.436213    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-openshift-ca" pod "e890ee22-58d5-495d-9cc3-ccf8fb863093" (UID: "e890ee22-58d5-495d-9cc3-ccf8fb863093"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:35.437968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.437928    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-xks-ca" pod "e890ee22-58d5-495d-9cc3-ccf8fb863093" (UID: "e890ee22-58d5-495d-9cc3-ccf8fb863093"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:35.438112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.438089    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e890ee22-58d5-495d-9cc3-ccf8fb863093-kube-api-access-dtpqd" pod "e890ee22-58d5-495d-9cc3-ccf8fb863093" (UID: "e890ee22-58d5-495d-9cc3-ccf8fb863093"). InnerVolumeSpecName "kube-api-access-dtpqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:35.520143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.520058    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"]
Oct 30 22:49:35.526660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.526619    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fb9ssnt"]
Oct 30 22:49:35.536520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.536497    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.536520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.536519    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e890ee22-58d5-495d-9cc3-ccf8fb863093-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.536688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.536529    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtpqd\" (UniqueName: \"kubernetes.io/projected/e890ee22-58d5-495d-9cc3-ccf8fb863093-kube-api-access-dtpqd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.734657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.734593    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:35.791610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:35.791529    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589be902-48e6-468a-bb48-94242a992854" path="/var/lib/kubelet/pods/589be902-48e6-468a-bb48-94242a992854/volumes"
Oct 30 22:49:36.240394 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.240360    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"
Oct 30 22:49:36.240895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.240368    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw" event={"ID":"e890ee22-58d5-495d-9cc3-ccf8fb863093","Type":"ContainerDied","Data":"0ae77bcc86d22f4f33f36d68fa38c349cfc3e8ca0e0e4d9bc762852174092701"}
Oct 30 22:49:36.240895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.240514    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ae77bcc86d22f4f33f36d68fa38c349cfc3e8ca0e0e4d9bc762852174092701"
Oct 30 22:49:36.242005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.241982    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh" event={"ID":"eea4a889-5455-4f4a-b278-31c0d2d45d84","Type":"ContainerDied","Data":"5bccc71754046d81801b8813b401ad49fe0f57e875ac221b34af858a87d0dc03"}
Oct 30 22:49:36.242145 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.242025    5503 scope.go:122] "RemoveContainer" containerID="e2efe4261aa23125a83c8fffdefdf882080a60ac16c72beab0ec864a8179fa37"
Oct 30 22:49:36.242221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.242174    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"
Oct 30 22:49:36.410945 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.410911    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"]
Oct 30 22:49:36.415436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:36.415410    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e65019/simulator-w1-e65019-kserve-68f88958d4-cq4nh"]
Oct 30 22:49:37.317748 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:37.317694    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:37.789139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:37.789105    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" path="/var/lib/kubelet/pods/eea4a889-5455-4f4a-b278-31c0d2d45d84/volumes"
Oct 30 22:49:38.851671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:38.851623    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"]
Oct 30 22:49:38.852152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:38.851890    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" containerID="cri-o://8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd" gracePeriod=30
Oct 30 22:49:40.329401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:40.329373    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:43.050947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:43.050909    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:49:43.051332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:43.050993    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:49:43.655842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:43.655791    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:43.822116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:43.822076    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:44.116575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.116549    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:49:44.224816 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.224736    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:49:44.224996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.224821    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:49:44.243259 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243230    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-socket\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243298    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-token\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-token\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243320    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-credential-socket\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243344    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-kube-api-access-pvtx5\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-kube-api-access-pvtx5\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243385    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-data\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243449    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istiod-ca-cert\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243495    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-podinfo\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243525    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-socket" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:44.243689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243618    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-envoy\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243678    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-certs\") pod \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\" (UID: \"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef\") "
Oct 30 22:49:44.243997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243690    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-credential-socket" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:44.243997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.243778    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-data" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:44.244114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.244096    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istiod-ca-cert" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:44.244187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.244109    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.244187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.244131    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.244187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.244146    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.244373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.244353    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-certs" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:44.250675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.246215    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-podinfo" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:44.250675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.246740    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-token" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:44.250675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.246767    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-kube-api-access-pvtx5" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "kube-api-access-pvtx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:44.250902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.250726    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-envoy" pod "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" (UID: "9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:44.303215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.303177    5503 generic.go:356] "Generic (PLEG): container finished" podID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerID="8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd" exitCode=0
Oct 30 22:49:44.303407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.303250    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"
Oct 30 22:49:44.303485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.303406    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" event={"ID":"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef","Type":"ContainerDied","Data":"8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd"}
Oct 30 22:49:44.303485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.303469    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b" event={"ID":"9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef","Type":"ContainerDied","Data":"f340609bdbe0df2d9f8fbb4e5d1901f9f53be60a8370245455a02e065a78698a"}
Oct 30 22:49:44.303605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.303499    5503 scope.go:122] "RemoveContainer" containerID="8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd"
Oct 30 22:49:44.318545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.318523    5503 scope.go:122] "RemoveContainer" containerID="8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd"
Oct 30 22:49:44.318910 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:49:44.318890    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd\": container with ID starting with 8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd not found: ID does not exist" containerID="8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd"
Oct 30 22:49:44.319026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.318917    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd"} err="failed to get container status \"8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd\": rpc error: code = NotFound desc = could not find container \"8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd\": container with ID starting with 8f46976df42bedaf90b5e12f60dbc6a75443cbfc8413527872798480df3081cd not found: ID does not exist"
Oct 30 22:49:44.344884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.344862    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.344884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.344886    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.344884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.344896    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.345143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.344910    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvtx5\" (UniqueName: \"kubernetes.io/projected/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-kube-api-access-pvtx5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.345143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.344923    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.345143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.344933    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:44.653626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.653093    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"]
Oct 30 22:49:44.660363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.660335    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-421286-openshift-default-84969cfcdf-xwx5b"]
Oct 30 22:49:44.828616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:44.828561    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:45.788946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:45.788910    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" path="/var/lib/kubelet/pods/9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef/volumes"
Oct 30 22:49:49.351561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:49.351526    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:50.776990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:50.776897    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:53.356888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:53.356845    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:54.810754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:54.810713    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:55.120509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:55.120400    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:55.278310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:55.278276    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:55.370880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:55.370805    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"]
Oct 30 22:49:55.421240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:55.421208    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b829bc-qllbc"]
Oct 30 22:49:55.789008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:55.788975    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f263e61-3d05-4325-9684-5d2e4e00dbc5" path="/var/lib/kubelet/pods/4f263e61-3d05-4325-9684-5d2e4e00dbc5/volumes"
Oct 30 22:49:55.821384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:55.821353    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:58.098486 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:49:58.098448    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:49:58.256003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:58.255966    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:58.843714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:58.843680    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:59.019152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:59.019101    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:49:59.167694 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:49:59.167659    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:00.159079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:00.159041    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:01.866036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:01.865995    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:50:02.446936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:02.446896    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:02.693310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:02.693271    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:02.853932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:02.853853    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:03.627397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.627361    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"]
Oct 30 22:50:03.628174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628156    5503 cpu_manager.go:414] "containerMap: removing container" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main"
Oct 30 22:50:03.628225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628177    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main"
Oct 30 22:50:03.628267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628236    5503 cpu_manager.go:414] "containerMap: removing container" podUID="589be902-48e6-468a-bb48-94242a992854" containerName="payload-processing"
Oct 30 22:50:03.628267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628245    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589be902-48e6-468a-bb48-94242a992854" containerName="payload-processing"
Oct 30 22:50:03.628267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628258    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy"
Oct 30 22:50:03.628363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628266    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy"
Oct 30 22:50:03.628363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628277    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e890ee22-58d5-495d-9cc3-ccf8fb863093" containerName="revoke-keys"
Oct 30 22:50:03.628363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628283    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e890ee22-58d5-495d-9cc3-ccf8fb863093" containerName="revoke-keys"
Oct 30 22:50:03.628467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628416    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="589be902-48e6-468a-bb48-94242a992854" containerName="payload-processing"
Oct 30 22:50:03.628467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628438    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df42c1d-bc6f-47f1-a75c-7b98f2fca6ef" containerName="istio-proxy"
Oct 30 22:50:03.628467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628448    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e890ee22-58d5-495d-9cc3-ccf8fb863093" containerName="revoke-keys"
Oct 30 22:50:03.628467 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.628464    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea4a889-5455-4f4a-b278-31c0d2d45d84" containerName="main"
Oct 30 22:50:03.633649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.633613    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.674797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.674761    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.674983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.674831    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7jv\" (UniqueName: \"kubernetes.io/projected/f12414c1-8181-4930-8e3e-c156cd78fbd2-kube-api-access-tf7jv\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.674983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.674900    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.684755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.684727    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"]
Oct 30 22:50:03.732364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.732321    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:03.732364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.732333    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:03.732675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.732377    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:03.737875 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.737850    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:50:03.776372 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.776339    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7jv\" (UniqueName: \"kubernetes.io/projected/f12414c1-8181-4930-8e3e-c156cd78fbd2-kube-api-access-tf7jv\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.776558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.776388    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.776616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.776599    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.777264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.777238    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.779019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.778994    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:03.888111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.888038    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:03.898214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:03.898189    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7jv\" (UniqueName: \"kubernetes.io/projected/f12414c1-8181-4930-8e3e-c156cd78fbd2-kube-api-access-tf7jv\") pod \"maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:04.001010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:04.000978    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:50:04.009189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:04.009169    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:04.381056 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:50:04.381021    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12414c1_8181_4930_8e3e_c156cd78fbd2.slice/crio-e8cd4ed333ccb6b3553df7ad6dfece6796c1c122eaaeba9a255d512ff1078426 WatchSource:0}: Error finding container e8cd4ed333ccb6b3553df7ad6dfece6796c1c122eaaeba9a255d512ff1078426: Status 404 returned error can't find the container with id e8cd4ed333ccb6b3553df7ad6dfece6796c1c122eaaeba9a255d512ff1078426
Oct 30 22:50:04.383872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:04.383821    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"]
Oct 30 22:50:04.432391 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:04.432360    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn" event={"ID":"f12414c1-8181-4930-8e3e-c156cd78fbd2","Type":"ContainerStarted","Data":"e8cd4ed333ccb6b3553df7ad6dfece6796c1c122eaaeba9a255d512ff1078426"}
Oct 30 22:50:05.128445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:05.128413    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:05.285812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:05.285774    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:05.440398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:05.440361    5503 generic.go:356] "Generic (PLEG): container finished" podID="f12414c1-8181-4930-8e3e-c156cd78fbd2" containerID="e5648a94ff890ca99eda9c8db232dc192206b0dc795d747460688e3ecc13fa4f" exitCode=0
Oct 30 22:50:05.440599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:05.440451    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn" event={"ID":"f12414c1-8181-4930-8e3e-c156cd78fbd2","Type":"ContainerDied","Data":"e5648a94ff890ca99eda9c8db232dc192206b0dc795d747460688e3ecc13fa4f"}
Oct 30 22:50:06.051017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.050982    5503 scope.go:122] "RemoveContainer" containerID="c9320bca6ce792117a33919f98ff32832457fe991d0a531e680d1ef96b4993b3"
Oct 30 22:50:06.062889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.062867    5503 scope.go:122] "RemoveContainer" containerID="36980df9f373bc27fb62aecdbceb00012b15ecfbfa36d3ca83161b269182a249"
Oct 30 22:50:06.590003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.589977    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:06.606757 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.606731    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-openshift-ca\") pod \"f12414c1-8181-4930-8e3e-c156cd78fbd2\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") "
Oct 30 22:50:06.606895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.606814    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-xks-ca\") pod \"f12414c1-8181-4930-8e3e-c156cd78fbd2\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") "
Oct 30 22:50:06.607050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.607028    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12414c1-8181-4930-8e3e-c156cd78fbd2-kube-api-access-tf7jv\" (UniqueName: \"kubernetes.io/projected/f12414c1-8181-4930-8e3e-c156cd78fbd2-kube-api-access-tf7jv\") pod \"f12414c1-8181-4930-8e3e-c156cd78fbd2\" (UID: \"f12414c1-8181-4930-8e3e-c156cd78fbd2\") "
Oct 30 22:50:06.607128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.607087    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-openshift-ca" pod "f12414c1-8181-4930-8e3e-c156cd78fbd2" (UID: "f12414c1-8181-4930-8e3e-c156cd78fbd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:06.607446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.607423    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.609010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.608977    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-xks-ca" pod "f12414c1-8181-4930-8e3e-c156cd78fbd2" (UID: "f12414c1-8181-4930-8e3e-c156cd78fbd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:06.609120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.609097    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12414c1-8181-4930-8e3e-c156cd78fbd2-kube-api-access-tf7jv" pod "f12414c1-8181-4930-8e3e-c156cd78fbd2" (UID: "f12414c1-8181-4930-8e3e-c156cd78fbd2"). InnerVolumeSpecName "kube-api-access-tf7jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:06.708142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.708074    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12414c1-8181-4930-8e3e-c156cd78fbd2-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:06.708142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:06.708100    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf7jv\" (UniqueName: \"kubernetes.io/projected/f12414c1-8181-4930-8e3e-c156cd78fbd2-kube-api-access-tf7jv\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:07.456487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:07.456457    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"
Oct 30 22:50:07.456689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:07.456489    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn" event={"ID":"f12414c1-8181-4930-8e3e-c156cd78fbd2","Type":"ContainerDied","Data":"e8cd4ed333ccb6b3553df7ad6dfece6796c1c122eaaeba9a255d512ff1078426"}
Oct 30 22:50:07.456689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:07.456535    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8cd4ed333ccb6b3553df7ad6dfece6796c1c122eaaeba9a255d512ff1078426"
Oct 30 22:50:10.015710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:10.015676    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:14.637849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:14.637810    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:15.245650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:15.245616    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"]
Oct 30 22:50:15.361407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:15.361368    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"]
Oct 30 22:50:15.630253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:15.630149    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:15.785518 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:15.785479    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:19.224788 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:19.224752    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:20.246888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.246829    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv" podUID="74a6a7fe-602d-4340-ab2c-1259b445e22f" containerName="payload-processing" containerID="cri-o://4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46" gracePeriod=25
Oct 30 22:50:20.362711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.362631    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt" podUID="187b3f91-b0e5-4837-afce-0904cd9a7732" containerName="payload-pre-processing" containerID="cri-o://a54f68442deed2ef921eed04bc5e37a6829c76301231dbac2f6a9056b3278cb4" gracePeriod=25
Oct 30 22:50:20.515854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.515798    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:50:20.533675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.533619    5503 generic.go:356] "Generic (PLEG): container finished" podID="74a6a7fe-602d-4340-ab2c-1259b445e22f" containerID="4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46" exitCode=0
Oct 30 22:50:20.533675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.533668    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv" event={"ID":"74a6a7fe-602d-4340-ab2c-1259b445e22f","Type":"ContainerDied","Data":"4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46"}
Oct 30 22:50:20.533902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.533680    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"
Oct 30 22:50:20.533902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.533730    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv" event={"ID":"74a6a7fe-602d-4340-ab2c-1259b445e22f","Type":"ContainerDied","Data":"689561ec0ca732b29a36c803fb232a10fe2bf972fc2ddc8facc9487604ec6c3b"}
Oct 30 22:50:20.533902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.533762    5503 scope.go:122] "RemoveContainer" containerID="4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46"
Oct 30 22:50:20.535556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.535531    5503 generic.go:356] "Generic (PLEG): container finished" podID="187b3f91-b0e5-4837-afce-0904cd9a7732" containerID="a54f68442deed2ef921eed04bc5e37a6829c76301231dbac2f6a9056b3278cb4" exitCode=0
Oct 30 22:50:20.535726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.535620    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt" event={"ID":"187b3f91-b0e5-4837-afce-0904cd9a7732","Type":"ContainerDied","Data":"a54f68442deed2ef921eed04bc5e37a6829c76301231dbac2f6a9056b3278cb4"}
Oct 30 22:50:20.545917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.545887    5503 scope.go:122] "RemoveContainer" containerID="4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46"
Oct 30 22:50:20.546198 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:50:20.546176    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46\": container with ID starting with 4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46 not found: ID does not exist" containerID="4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46"
Oct 30 22:50:20.546263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.546205    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46"} err="failed to get container status \"4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46\": rpc error: code = NotFound desc = could not find container \"4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46\": container with ID starting with 4fab3317cf865d58d87f5c5467e35a28b8c1f60ac61094ba9cdbab195e3b6f46 not found: ID does not exist"
Oct 30 22:50:20.646057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.646030    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:50:20.666937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.666913    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a6a7fe-602d-4340-ab2c-1259b445e22f-kube-api-access-6tbfd\" (UniqueName: \"kubernetes.io/projected/74a6a7fe-602d-4340-ab2c-1259b445e22f-kube-api-access-6tbfd\") pod \"74a6a7fe-602d-4340-ab2c-1259b445e22f\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") "
Oct 30 22:50:20.667150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.667132    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a6a7fe-602d-4340-ab2c-1259b445e22f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a6a7fe-602d-4340-ab2c-1259b445e22f-plugins-config-volume\") pod \"74a6a7fe-602d-4340-ab2c-1259b445e22f\" (UID: \"74a6a7fe-602d-4340-ab2c-1259b445e22f\") "
Oct 30 22:50:20.667465 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.667438    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a6a7fe-602d-4340-ab2c-1259b445e22f-plugins-config-volume" pod "74a6a7fe-602d-4340-ab2c-1259b445e22f" (UID: "74a6a7fe-602d-4340-ab2c-1259b445e22f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:20.667598 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.667584    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a6a7fe-602d-4340-ab2c-1259b445e22f-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.668965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.668947    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a6a7fe-602d-4340-ab2c-1259b445e22f-kube-api-access-6tbfd" pod "74a6a7fe-602d-4340-ab2c-1259b445e22f" (UID: "74a6a7fe-602d-4340-ab2c-1259b445e22f"). InnerVolumeSpecName "kube-api-access-6tbfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:20.768462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.768342    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187b3f91-b0e5-4837-afce-0904cd9a7732-kube-api-access-bd6sd\" (UniqueName: \"kubernetes.io/projected/187b3f91-b0e5-4837-afce-0904cd9a7732-kube-api-access-bd6sd\") pod \"187b3f91-b0e5-4837-afce-0904cd9a7732\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") "
Oct 30 22:50:20.780026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.768485    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187b3f91-b0e5-4837-afce-0904cd9a7732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b3f91-b0e5-4837-afce-0904cd9a7732-plugins-config-volume\") pod \"187b3f91-b0e5-4837-afce-0904cd9a7732\" (UID: \"187b3f91-b0e5-4837-afce-0904cd9a7732\") "
Oct 30 22:50:20.780026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.768757    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tbfd\" (UniqueName: \"kubernetes.io/projected/74a6a7fe-602d-4340-ab2c-1259b445e22f-kube-api-access-6tbfd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.780026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.768858    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187b3f91-b0e5-4837-afce-0904cd9a7732-plugins-config-volume" pod "187b3f91-b0e5-4837-afce-0904cd9a7732" (UID: "187b3f91-b0e5-4837-afce-0904cd9a7732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:20.780026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.770491    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187b3f91-b0e5-4837-afce-0904cd9a7732-kube-api-access-bd6sd" pod "187b3f91-b0e5-4837-afce-0904cd9a7732" (UID: "187b3f91-b0e5-4837-afce-0904cd9a7732"). InnerVolumeSpecName "kube-api-access-bd6sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:20.869879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.869849    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187b3f91-b0e5-4837-afce-0904cd9a7732-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.869879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.869881    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd6sd\" (UniqueName: \"kubernetes.io/projected/187b3f91-b0e5-4837-afce-0904cd9a7732-kube-api-access-bd6sd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.957399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.957364    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"]
Oct 30 22:50:20.961864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:20.961837    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f4e241-c9f759785-2sqcv"]
Oct 30 22:50:21.543594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:21.543511    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"
Oct 30 22:50:21.543594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:21.543549    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt" event={"ID":"187b3f91-b0e5-4837-afce-0904cd9a7732","Type":"ContainerDied","Data":"421f1f5f763e8062f234a89d8b307b10dfe342029b6cce60017231ed4e8915f2"}
Oct 30 22:50:21.544179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:21.543608    5503 scope.go:122] "RemoveContainer" containerID="a54f68442deed2ef921eed04bc5e37a6829c76301231dbac2f6a9056b3278cb4"
Oct 30 22:50:21.717069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:21.717034    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"]
Oct 30 22:50:21.789368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:21.789336    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a6a7fe-602d-4340-ab2c-1259b445e22f" path="/var/lib/kubelet/pods/74a6a7fe-602d-4340-ab2c-1259b445e22f/volumes"
Oct 30 22:50:21.789692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:21.789677    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-f4e241-695766bf9-2wfvt"]
Oct 30 22:50:23.751533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:23.751497    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:23.792591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:23.792557    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187b3f91-b0e5-4837-afce-0904cd9a7732" path="/var/lib/kubelet/pods/187b3f91-b0e5-4837-afce-0904cd9a7732/volumes"
Oct 30 22:50:28.098647 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:50:28.098600    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:50:28.291289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:28.291241    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:31.319080 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:31.319042    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:32.575491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:32.575458    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:32.819522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:32.819487    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:37.315315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:37.315280    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:40.222575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:40.222542    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:50:41.843768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:41.843690    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:45.812050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812018    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"]
Oct 30 22:50:45.812520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812506    5503 cpu_manager.go:414] "containerMap: removing container" podUID="f12414c1-8181-4930-8e3e-c156cd78fbd2" containerName="revoke-keys"
Oct 30 22:50:45.812567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812521    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12414c1-8181-4930-8e3e-c156cd78fbd2" containerName="revoke-keys"
Oct 30 22:50:45.812567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812535    5503 cpu_manager.go:414] "containerMap: removing container" podUID="74a6a7fe-602d-4340-ab2c-1259b445e22f" containerName="payload-processing"
Oct 30 22:50:45.812567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812541    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a6a7fe-602d-4340-ab2c-1259b445e22f" containerName="payload-processing"
Oct 30 22:50:45.812567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812555    5503 cpu_manager.go:414] "containerMap: removing container" podUID="187b3f91-b0e5-4837-afce-0904cd9a7732" containerName="payload-pre-processing"
Oct 30 22:50:45.812567 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812560    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187b3f91-b0e5-4837-afce-0904cd9a7732" containerName="payload-pre-processing"
Oct 30 22:50:45.812814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812666    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12414c1-8181-4930-8e3e-c156cd78fbd2" containerName="revoke-keys"
Oct 30 22:50:45.812814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812676    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a6a7fe-602d-4340-ab2c-1259b445e22f" containerName="payload-processing"
Oct 30 22:50:45.813157 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.812685    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="187b3f91-b0e5-4837-afce-0904cd9a7732" containerName="payload-pre-processing"
Oct 30 22:50:45.818347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.818327    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:45.866719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.866684    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"]
Oct 30 22:50:45.870862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.870839    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:45.873233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.873210    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"]
Oct 30 22:50:45.915149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.915101    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:45.915349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.915316    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:50:45.921826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.921801    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:50:45.921826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.921821    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:50:45.956142 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.956118    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:45.956267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.956167    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2h5v\" (UniqueName: \"kubernetes.io/projected/334853a1-ed1a-44c4-8964-84a7f51c4ca8-kube-api-access-p2h5v\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:45.956267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.956221    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2x4c\" (UniqueName: \"kubernetes.io/projected/562effb4-26cd-462a-898f-3cb93e112042-kube-api-access-v2x4c\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:45.956350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.956325    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562effb4-26cd-462a-898f-3cb93e112042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:45.956388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.956355    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:45.956427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.956385    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562effb4-26cd-462a-898f-3cb93e112042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:45.965971 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:45.965945    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"]
Oct 30 22:50:46.057261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.057232    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2h5v\" (UniqueName: \"kubernetes.io/projected/334853a1-ed1a-44c4-8964-84a7f51c4ca8-kube-api-access-p2h5v\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:46.057446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.057270    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2x4c\" (UniqueName: \"kubernetes.io/projected/562effb4-26cd-462a-898f-3cb93e112042-kube-api-access-v2x4c\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:46.057446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.057353    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562effb4-26cd-462a-898f-3cb93e112042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:46.057446 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.057418    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:46.057584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.057465    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562effb4-26cd-462a-898f-3cb93e112042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:46.057584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.057579    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:46.058315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.058293    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562effb4-26cd-462a-898f-3cb93e112042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:46.058315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.058305    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:46.059836 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.059817    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:46.059926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.059909    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562effb4-26cd-462a-898f-3cb93e112042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:46.119909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.119834    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2h5v\" (UniqueName: \"kubernetes.io/projected/334853a1-ed1a-44c4-8964-84a7f51c4ca8-kube-api-access-p2h5v\") pod \"maas-api-revoke-sub-3adac6e5-4ck4g\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:46.119909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.119897    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2x4c\" (UniqueName: \"kubernetes.io/projected/562effb4-26cd-462a-898f-3cb93e112042-kube-api-access-v2x4c\") pod \"maas-api-revoke-sub-3812fa8d-r7z4k\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:46.133537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.133516    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:46.187845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.187811    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:46.501983 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:50:46.501948    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334853a1_ed1a_44c4_8964_84a7f51c4ca8.slice/crio-6cfb2c213707554ff2adea8c9a12ce3054e6b210f6cb57a3bf8df6bc99f0f141 WatchSource:0}: Error finding container 6cfb2c213707554ff2adea8c9a12ce3054e6b210f6cb57a3bf8df6bc99f0f141: Status 404 returned error can't find the container with id 6cfb2c213707554ff2adea8c9a12ce3054e6b210f6cb57a3bf8df6bc99f0f141
Oct 30 22:50:46.504431 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.504408    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"]
Oct 30 22:50:46.596240 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:50:46.596208    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562effb4_26cd_462a_898f_3cb93e112042.slice/crio-55861adab09175bc26bc863faf87b50ab8e70fbadb27892151f1a2d5c2c26658 WatchSource:0}: Error finding container 55861adab09175bc26bc863faf87b50ab8e70fbadb27892151f1a2d5c2c26658: Status 404 returned error can't find the container with id 55861adab09175bc26bc863faf87b50ab8e70fbadb27892151f1a2d5c2c26658
Oct 30 22:50:46.598360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.598337    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"]
Oct 30 22:50:46.631866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.631841    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:46.696222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.696185    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g" event={"ID":"334853a1-ed1a-44c4-8964-84a7f51c4ca8","Type":"ContainerStarted","Data":"6cfb2c213707554ff2adea8c9a12ce3054e6b210f6cb57a3bf8df6bc99f0f141"}
Oct 30 22:50:46.697421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:46.697393    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k" event={"ID":"562effb4-26cd-462a-898f-3cb93e112042","Type":"ContainerStarted","Data":"55861adab09175bc26bc863faf87b50ab8e70fbadb27892151f1a2d5c2c26658"}
Oct 30 22:50:47.704974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:47.704878    5503 generic.go:356] "Generic (PLEG): container finished" podID="562effb4-26cd-462a-898f-3cb93e112042" containerID="84d3e49a9a1cbb1143efea221bae14006c65adf14e7c6a0c3ba271b49bc14f6c" exitCode=0
Oct 30 22:50:47.704974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:47.704959    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k" event={"ID":"562effb4-26cd-462a-898f-3cb93e112042","Type":"ContainerDied","Data":"84d3e49a9a1cbb1143efea221bae14006c65adf14e7c6a0c3ba271b49bc14f6c"}
Oct 30 22:50:47.706599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:47.706573    5503 generic.go:356] "Generic (PLEG): container finished" podID="334853a1-ed1a-44c4-8964-84a7f51c4ca8" containerID="4181c316b3d60f3662b4b3d591f88a1f2b8bf05e4faf1d4b45a13596db38ffd2" exitCode=0
Oct 30 22:50:47.706696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:47.706598    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g" event={"ID":"334853a1-ed1a-44c4-8964-84a7f51c4ca8","Type":"ContainerDied","Data":"4181c316b3d60f3662b4b3d591f88a1f2b8bf05e4faf1d4b45a13596db38ffd2"}
Oct 30 22:50:48.888537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:48.888514    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:48.891662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:48.891628    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:49.000345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.000269    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562effb4-26cd-462a-898f-3cb93e112042-kube-api-access-v2x4c\" (UniqueName: \"kubernetes.io/projected/562effb4-26cd-462a-898f-3cb93e112042-kube-api-access-v2x4c\") pod \"562effb4-26cd-462a-898f-3cb93e112042\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") "
Oct 30 22:50:49.000345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.000319    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/562effb4-26cd-462a-898f-3cb93e112042-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562effb4-26cd-462a-898f-3cb93e112042-maas-api-xks-ca\") pod \"562effb4-26cd-462a-898f-3cb93e112042\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") "
Oct 30 22:50:49.000549 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.000377    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-openshift-ca\") pod \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") "
Oct 30 22:50:49.000549 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.000502    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334853a1-ed1a-44c4-8964-84a7f51c4ca8-kube-api-access-p2h5v\" (UniqueName: \"kubernetes.io/projected/334853a1-ed1a-44c4-8964-84a7f51c4ca8-kube-api-access-p2h5v\") pod \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") "
Oct 30 22:50:49.000690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.000673    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-xks-ca\") pod \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\" (UID: \"334853a1-ed1a-44c4-8964-84a7f51c4ca8\") "
Oct 30 22:50:49.000761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.000705    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/562effb4-26cd-462a-898f-3cb93e112042-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562effb4-26cd-462a-898f-3cb93e112042-maas-api-openshift-ca\") pod \"562effb4-26cd-462a-898f-3cb93e112042\" (UID: \"562effb4-26cd-462a-898f-3cb93e112042\") "
Oct 30 22:50:49.000856 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.000831    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-openshift-ca" pod "334853a1-ed1a-44c4-8964-84a7f51c4ca8" (UID: "334853a1-ed1a-44c4-8964-84a7f51c4ca8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:49.001143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.001117    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562effb4-26cd-462a-898f-3cb93e112042-maas-api-openshift-ca" pod "562effb4-26cd-462a-898f-3cb93e112042" (UID: "562effb4-26cd-462a-898f-3cb93e112042"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:49.001281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.001164    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:49.003180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.003144    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562effb4-26cd-462a-898f-3cb93e112042-kube-api-access-v2x4c" pod "562effb4-26cd-462a-898f-3cb93e112042" (UID: "562effb4-26cd-462a-898f-3cb93e112042"). InnerVolumeSpecName "kube-api-access-v2x4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:49.003311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.003267    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334853a1-ed1a-44c4-8964-84a7f51c4ca8-kube-api-access-p2h5v" pod "334853a1-ed1a-44c4-8964-84a7f51c4ca8" (UID: "334853a1-ed1a-44c4-8964-84a7f51c4ca8"). InnerVolumeSpecName "kube-api-access-p2h5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:49.003651 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.003610    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/562effb4-26cd-462a-898f-3cb93e112042-maas-api-xks-ca" pod "562effb4-26cd-462a-898f-3cb93e112042" (UID: "562effb4-26cd-462a-898f-3cb93e112042"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:49.003805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.003772    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-xks-ca" pod "334853a1-ed1a-44c4-8964-84a7f51c4ca8" (UID: "334853a1-ed1a-44c4-8964-84a7f51c4ca8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:49.102313 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.102280    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2h5v\" (UniqueName: \"kubernetes.io/projected/334853a1-ed1a-44c4-8964-84a7f51c4ca8-kube-api-access-p2h5v\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:49.102313 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.102306    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334853a1-ed1a-44c4-8964-84a7f51c4ca8-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:49.102493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.102320    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/562effb4-26cd-462a-898f-3cb93e112042-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:49.102493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.102334    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2x4c\" (UniqueName: \"kubernetes.io/projected/562effb4-26cd-462a-898f-3cb93e112042-kube-api-access-v2x4c\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:49.102493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.102343    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/562effb4-26cd-462a-898f-3cb93e112042-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:49.719808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.719773    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"
Oct 30 22:50:49.719808 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.719796    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k" event={"ID":"562effb4-26cd-462a-898f-3cb93e112042","Type":"ContainerDied","Data":"55861adab09175bc26bc863faf87b50ab8e70fbadb27892151f1a2d5c2c26658"}
Oct 30 22:50:49.720070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.719892    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55861adab09175bc26bc863faf87b50ab8e70fbadb27892151f1a2d5c2c26658"
Oct 30 22:50:49.721207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.721187    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"
Oct 30 22:50:49.721341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.721234    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g" event={"ID":"334853a1-ed1a-44c4-8964-84a7f51c4ca8","Type":"ContainerDied","Data":"6cfb2c213707554ff2adea8c9a12ce3054e6b210f6cb57a3bf8df6bc99f0f141"}
Oct 30 22:50:49.721341 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:49.721275    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cfb2c213707554ff2adea8c9a12ce3054e6b210f6cb57a3bf8df6bc99f0f141"
Oct 30 22:50:51.066309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:51.066264    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:51.224387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:51.224357    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:51.386940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:51.386864    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:55.750171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:55.750136    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:50:58.098576 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:50:58.098541    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:50:59.801140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:50:59.801105    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:01.261754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:01.261715    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:01.519454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:01.519364    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:01.695594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:01.695558    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:01.775264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:01.775178    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"]
Oct 30 22:51:01.825209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:01.825170    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9a60dd-stqhw"]
Oct 30 22:51:02.187832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:02.187794    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:02.429391 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:02.429349    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:02.754369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:02.754334    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:03.789777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:03.789747    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e890ee22-58d5-495d-9cc3-ccf8fb863093" path="/var/lib/kubelet/pods/e890ee22-58d5-495d-9cc3-ccf8fb863093/volumes"
Oct 30 22:51:06.426747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:06.426659    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:07.938619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:07.938582    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.100077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:08.100041    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.260397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:08.260318    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:08.831328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:08.831294    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:09.034655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.034605    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:09.232329 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.232298    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"]
Oct 30 22:51:09.232532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.232508    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy" containerID="cri-o://02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6" gracePeriod=30
Oct 30 22:51:09.343032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.342995    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:09.474199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.474160    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"]
Oct 30 22:51:09.474943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.474918    5503 cpu_manager.go:414] "containerMap: removing container" podUID="334853a1-ed1a-44c4-8964-84a7f51c4ca8" containerName="revoke-keys"
Oct 30 22:51:09.474943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.474942    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334853a1-ed1a-44c4-8964-84a7f51c4ca8" containerName="revoke-keys"
Oct 30 22:51:09.475144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.474962    5503 cpu_manager.go:414] "containerMap: removing container" podUID="562effb4-26cd-462a-898f-3cb93e112042" containerName="revoke-keys"
Oct 30 22:51:09.475144 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.474970    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="562effb4-26cd-462a-898f-3cb93e112042" containerName="revoke-keys"
Oct 30 22:51:09.475243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.475165    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="334853a1-ed1a-44c4-8964-84a7f51c4ca8" containerName="revoke-keys"
Oct 30 22:51:09.475243 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.475183    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="562effb4-26cd-462a-898f-3cb93e112042" containerName="revoke-keys"
Oct 30 22:51:09.479221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.479194    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.533550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.533520    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"]
Oct 30 22:51:09.552475 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.552442    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7ml\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-kube-api-access-4k7ml\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.552698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.552506    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.552698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.552653    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.552842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.552694    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.653991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.653945    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7ml\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-kube-api-access-4k7ml\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.654188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.654008    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.654188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.654070    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.654188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.654101    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.654706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.654681    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.654881 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.654859    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.656775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.656747    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.716397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.716367    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7ml\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-kube-api-access-4k7ml\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.796734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.796660    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:09.840862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:09.840793    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:10.061909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.061842    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"]
Oct 30 22:51:10.066720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.066702    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.119679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.119631    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"]
Oct 30 22:51:10.120185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.120160    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:10.120269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.120252    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:51:10.120893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.120869    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:10.120893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.120877    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:10.159711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.159683    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.159832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.159805    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.159943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.159919    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdm4\" (UniqueName: \"kubernetes.io/projected/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-kube-api-access-xxdm4\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.167611 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:51:10.167580    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2a81d9_4a8a_47a1_834d_c6e0abd15785.slice/crio-99efa1368d94f50dd49e7da14efbc7ae2abda7ff1c870539022cc84b6b80c060 WatchSource:0}: Error finding container 99efa1368d94f50dd49e7da14efbc7ae2abda7ff1c870539022cc84b6b80c060: Status 404 returned error can't find the container with id 99efa1368d94f50dd49e7da14efbc7ae2abda7ff1c870539022cc84b6b80c060
Oct 30 22:51:10.170188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.170165    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"]
Oct 30 22:51:10.260765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.260735    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.260932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.260800    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.260932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.260857    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdm4\" (UniqueName: \"kubernetes.io/projected/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-kube-api-access-xxdm4\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.261426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.261400    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.263030 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.263012    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.323194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.323130    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdm4\" (UniqueName: \"kubernetes.io/projected/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-kube-api-access-xxdm4\") pod \"maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.379859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.379824    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:10.747559 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:51:10.747535    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0881fb93_59a0_4ea8_a982_5cc8e2f83b80.slice/crio-1db41b223ca60295c1815c537523f3d3a6866b858c3b7840b2593abc5dfc8b50 WatchSource:0}: Error finding container 1db41b223ca60295c1815c537523f3d3a6866b858c3b7840b2593abc5dfc8b50: Status 404 returned error can't find the container with id 1db41b223ca60295c1815c537523f3d3a6866b858c3b7840b2593abc5dfc8b50
Oct 30 22:51:10.750043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.750024    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"]
Oct 30 22:51:10.853535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.853503    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7" event={"ID":"0881fb93-59a0-4ea8-a982-5cc8e2f83b80","Type":"ContainerStarted","Data":"1db41b223ca60295c1815c537523f3d3a6866b858c3b7840b2593abc5dfc8b50"}
Oct 30 22:51:10.855014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.854986    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" event={"ID":"6e2a81d9-4a8a-47a1-834d-c6e0abd15785","Type":"ContainerStarted","Data":"79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7"}
Oct 30 22:51:10.855161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.855039    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" event={"ID":"6e2a81d9-4a8a-47a1-834d-c6e0abd15785","Type":"ContainerStarted","Data":"99efa1368d94f50dd49e7da14efbc7ae2abda7ff1c870539022cc84b6b80c060"}
Oct 30 22:51:10.855161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.855098    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:10.973751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:10.973695    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" podStartSLOduration=1.9736771819999999 podStartE2EDuration="1.973677182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:10.973153017 +0000 UTC m=+3847.307729787" watchObservedRunningTime="2026-10-30 22:51:10.973677182 +0000 UTC m=+3847.308253952"
Oct 30 22:51:11.310829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:11.310719    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:11.863412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:11.863363    5503 generic.go:356] "Generic (PLEG): container finished" podID="0881fb93-59a0-4ea8-a982-5cc8e2f83b80" containerID="bd701a5248886b641e5f7047296484889ff372c07e72ac84d78f070b8932ef6b" exitCode=0
Oct 30 22:51:11.863616 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:11.863419    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7" event={"ID":"0881fb93-59a0-4ea8-a982-5cc8e2f83b80","Type":"ContainerDied","Data":"bd701a5248886b641e5f7047296484889ff372c07e72ac84d78f070b8932ef6b"}
Oct 30 22:51:12.821774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:12.821742    5503 patch_prober.go:29] interesting pod/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:51:12.822180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:12.821794    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:51:13.014041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.014015    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:13.196533 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.196488    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-openshift-ca\") pod \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") "
Oct 30 22:51:13.196775 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.196544    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-kube-api-access-xxdm4\" (UniqueName: \"kubernetes.io/projected/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-kube-api-access-xxdm4\") pod \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") "
Oct 30 22:51:13.196858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.196829    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-xks-ca\") pod \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\" (UID: \"0881fb93-59a0-4ea8-a982-5cc8e2f83b80\") "
Oct 30 22:51:13.196952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.196927    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-openshift-ca" pod "0881fb93-59a0-4ea8-a982-5cc8e2f83b80" (UID: "0881fb93-59a0-4ea8-a982-5cc8e2f83b80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:13.197208 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.197186    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:13.198728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.198704    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-kube-api-access-xxdm4" pod "0881fb93-59a0-4ea8-a982-5cc8e2f83b80" (UID: "0881fb93-59a0-4ea8-a982-5cc8e2f83b80"). InnerVolumeSpecName "kube-api-access-xxdm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:13.198821 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.198736    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-xks-ca" pod "0881fb93-59a0-4ea8-a982-5cc8e2f83b80" (UID: "0881fb93-59a0-4ea8-a982-5cc8e2f83b80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:13.298360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.298331    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:13.298360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.298359    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdm4\" (UniqueName: \"kubernetes.io/projected/0881fb93-59a0-4ea8-a982-5cc8e2f83b80-kube-api-access-xxdm4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:13.878261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.878229    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"
Oct 30 22:51:13.878704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.878259    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7" event={"ID":"0881fb93-59a0-4ea8-a982-5cc8e2f83b80","Type":"ContainerDied","Data":"1db41b223ca60295c1815c537523f3d3a6866b858c3b7840b2593abc5dfc8b50"}
Oct 30 22:51:13.878704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:13.878333    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db41b223ca60295c1815c537523f3d3a6866b858c3b7840b2593abc5dfc8b50"
Oct 30 22:51:14.483853 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.483831    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:51:14.613181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613103    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-envoy\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613200    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-data\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613302    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-certs\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613334    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-token\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-token\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613401    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-socket\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613439    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c163a492-0c0a-4664-805b-5798e4cd42a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c163a492-0c0a-4664-805b-5798e4cd42a2-istiod-ca-cert\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613470    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-podinfo\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613512    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-credential-socket\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.613815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613591    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-kube-api-access-ts6zh\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-kube-api-access-ts6zh\") pod \"c163a492-0c0a-4664-805b-5798e4cd42a2\" (UID: \"c163a492-0c0a-4664-805b-5798e4cd42a2\") "
Oct 30 22:51:14.614414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.613979    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-data" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:14.614414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.614179    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-socket" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:14.614414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.614203    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c163a492-0c0a-4664-805b-5798e4cd42a2-istiod-ca-cert" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:14.614414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.614384    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-certs" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:14.615010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.614985    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-credential-socket" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:14.616807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.616776    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-token" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:14.616980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.616957    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-kube-api-access-ts6zh" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "kube-api-access-ts6zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:14.617071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.616997    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-envoy" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:14.617992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.617972    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-podinfo" pod "c163a492-0c0a-4664-805b-5798e4cd42a2" (UID: "c163a492-0c0a-4664-805b-5798e4cd42a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:51:14.715524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715492    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715520    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715529    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715538    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715546    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c163a492-0c0a-4664-805b-5798e4cd42a2-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715554    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715571    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715580    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6zh\" (UniqueName: \"kubernetes.io/projected/c163a492-0c0a-4664-805b-5798e4cd42a2-kube-api-access-ts6zh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.715783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.715591    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c163a492-0c0a-4664-805b-5798e4cd42a2-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:14.886547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.886455    5503 generic.go:356] "Generic (PLEG): container finished" podID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerID="02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6" exitCode=0
Oct 30 22:51:14.886547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.886510    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"
Oct 30 22:51:14.887031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.886540    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" event={"ID":"c163a492-0c0a-4664-805b-5798e4cd42a2","Type":"ContainerDied","Data":"02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6"}
Oct 30 22:51:14.887031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.886597    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql" event={"ID":"c163a492-0c0a-4664-805b-5798e4cd42a2","Type":"ContainerDied","Data":"1161a14f9d1682b86ddc32f4dcf148c22fd51d011fd44c4caebee4b253b4c413"}
Oct 30 22:51:14.887031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.886619    5503 scope.go:122] "RemoveContainer" containerID="02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6"
Oct 30 22:51:14.897441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.897423    5503 scope.go:122] "RemoveContainer" containerID="a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2"
Oct 30 22:51:14.909207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.909185    5503 scope.go:122] "RemoveContainer" containerID="02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6"
Oct 30 22:51:14.909525 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:14.909502    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6\": container with ID starting with 02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6 not found: ID does not exist" containerID="02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6"
Oct 30 22:51:14.909570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.909534    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6"} err="failed to get container status \"02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6\": rpc error: code = NotFound desc = could not find container \"02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6\": container with ID starting with 02c316d3b4f688872f9dcefcf39cd45058f6ac9b7026aef121a9f783ad9db7f6 not found: ID does not exist"
Oct 30 22:51:14.909570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.909552    5503 scope.go:122] "RemoveContainer" containerID="a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2"
Oct 30 22:51:14.909854 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:14.909835    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2\": container with ID starting with a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2 not found: ID does not exist" containerID="a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2"
Oct 30 22:51:14.909917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:14.909860    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2"} err="failed to get container status \"a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2\": rpc error: code = NotFound desc = could not find container \"a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2\": container with ID starting with a250325a2e04221c475900ee5aad652cd1d41002d928e8731550975888d7fad2 not found: ID does not exist"
Oct 30 22:51:15.056090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:15.056057    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"]
Oct 30 22:51:15.106439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:15.106411    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9a60dd-openshift-default-8d7648469-6nfql"]
Oct 30 22:51:15.485139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:15.485089    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:15.757220 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:15.757133    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:15.789713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:15.789683    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" path="/var/lib/kubelet/pods/c163a492-0c0a-4664-805b-5798e4cd42a2/volumes"
Oct 30 22:51:16.225597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:16.225558    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:20.974827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:20.974784    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:21.158820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:21.158786    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:21.331191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:21.331104    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:21.866702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:21.866655    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" podUID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:22.518125 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.518088    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:51:22.930187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.930153    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:22.960979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.960941    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"]
Oct 30 22:51:22.962880 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.962849    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" podUID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:22.995005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.994970    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"]
Oct 30 22:51:22.995483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995463    5503 cpu_manager.go:414] "containerMap: removing container" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy"
Oct 30 22:51:22.995483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995479    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy"
Oct 30 22:51:22.995483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995488    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0881fb93-59a0-4ea8-a982-5cc8e2f83b80" containerName="revoke-keys"
Oct 30 22:51:22.995817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995493    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0881fb93-59a0-4ea8-a982-5cc8e2f83b80" containerName="revoke-keys"
Oct 30 22:51:22.995817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995508    5503 cpu_manager.go:414] "containerMap: removing container" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy"
Oct 30 22:51:22.995817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995513    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy"
Oct 30 22:51:22.995817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995615    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy"
Oct 30 22:51:22.995817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995628    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0881fb93-59a0-4ea8-a982-5cc8e2f83b80" containerName="revoke-keys"
Oct 30 22:51:22.995817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:22.995655    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="c163a492-0c0a-4664-805b-5798e4cd42a2" containerName="istio-proxy"
Oct 30 22:51:23.002241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.002212    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.005498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.005463    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.005498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.005499    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgvkw\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-kube-api-access-zgvkw\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.005755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.005527    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.005847 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.005815    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.064399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.064370    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"]
Oct 30 22:51:23.107095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.107065    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.107095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.107100    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgvkw\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-kube-api-access-zgvkw\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.107348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.107124    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.107348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.107220    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.107869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.107836    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.108037 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.108017    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.109593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.109571    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.169675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.169627    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgvkw\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-kube-api-access-zgvkw\") pod \"payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") " pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.326262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.326151    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:23.714613 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:51:23.714580    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc35d222_4674_4ffd_8373_de52d20deb4e.slice/crio-478dc7e5c2098b69602fca7e47e008f63f5074ba44cb644ceb6856fb0a824034 WatchSource:0}: Error finding container 478dc7e5c2098b69602fca7e47e008f63f5074ba44cb644ceb6856fb0a824034: Status 404 returned error can't find the container with id 478dc7e5c2098b69602fca7e47e008f63f5074ba44cb644ceb6856fb0a824034
Oct 30 22:51:23.717162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.717138    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"]
Oct 30 22:51:23.945588 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.945550    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" event={"ID":"fc35d222-4674-4ffd-8373-de52d20deb4e","Type":"ContainerStarted","Data":"58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2"}
Oct 30 22:51:23.945783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.945614    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" event={"ID":"fc35d222-4674-4ffd-8373-de52d20deb4e","Type":"ContainerStarted","Data":"478dc7e5c2098b69602fca7e47e008f63f5074ba44cb644ceb6856fb0a824034"}
Oct 30 22:51:23.945783 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:23.945674    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:51:24.065222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:24.065127    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" podStartSLOduration=2.065114823 podStartE2EDuration="2.065114823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:24.06366904 +0000 UTC m=+3860.398245807" watchObservedRunningTime="2026-10-30 22:51:24.065114823 +0000 UTC m=+3860.399691592"
Oct 30 22:51:24.846307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:24.846272    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:25.000445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:25.000414    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:25.624720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:25.624677    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:27.961890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:27.961838    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" podUID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" containerName="payload-processing" containerID="cri-o://79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7" gracePeriod=25
Oct 30 22:51:28.099997 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:28.099956    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:51:28.211860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.211834    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:28.272760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.272676    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-provider-credentials\") pod \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") "
Oct 30 22:51:28.272914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.272776    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-routing-overlay-volume\") pod \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") "
Oct 30 22:51:28.272969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.272954    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-plugins-config-volume\") pod \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") "
Oct 30 22:51:28.273024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.273009    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-kube-api-access-4k7ml\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-kube-api-access-4k7ml\") pod \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\" (UID: \"6e2a81d9-4a8a-47a1-834d-c6e0abd15785\") "
Oct 30 22:51:28.273167 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.273141    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-routing-overlay-volume" pod "6e2a81d9-4a8a-47a1-834d-c6e0abd15785" (UID: "6e2a81d9-4a8a-47a1-834d-c6e0abd15785"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:28.273398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.273367    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-plugins-config-volume" pod "6e2a81d9-4a8a-47a1-834d-c6e0abd15785" (UID: "6e2a81d9-4a8a-47a1-834d-c6e0abd15785"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:28.273540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.273494    5503 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-routing-overlay-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:28.273540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.273517    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:28.274976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.274953    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-provider-credentials" pod "6e2a81d9-4a8a-47a1-834d-c6e0abd15785" (UID: "6e2a81d9-4a8a-47a1-834d-c6e0abd15785"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:28.275231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.275216    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-kube-api-access-4k7ml" pod "6e2a81d9-4a8a-47a1-834d-c6e0abd15785" (UID: "6e2a81d9-4a8a-47a1-834d-c6e0abd15785"). InnerVolumeSpecName "kube-api-access-4k7ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:28.375000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.374958    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k7ml\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-kube-api-access-4k7ml\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:28.375000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.374991    5503 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e2a81d9-4a8a-47a1-834d-c6e0abd15785-provider-credentials\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:28.890823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.890782    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:28.976900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.976860    5503 generic.go:356] "Generic (PLEG): container finished" podID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" containerID="79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7" exitCode=0
Oct 30 22:51:28.977366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.976917    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"
Oct 30 22:51:28.977366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.976948    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" event={"ID":"6e2a81d9-4a8a-47a1-834d-c6e0abd15785","Type":"ContainerDied","Data":"79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7"}
Oct 30 22:51:28.977366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.977004    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76" event={"ID":"6e2a81d9-4a8a-47a1-834d-c6e0abd15785","Type":"ContainerDied","Data":"99efa1368d94f50dd49e7da14efbc7ae2abda7ff1c870539022cc84b6b80c060"}
Oct 30 22:51:28.977366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.977030    5503 scope.go:122] "RemoveContainer" containerID="79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7"
Oct 30 22:51:28.987203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.987180    5503 scope.go:122] "RemoveContainer" containerID="79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7"
Oct 30 22:51:28.987454 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:28.987434    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7\": container with ID starting with 79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7 not found: ID does not exist" containerID="79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7"
Oct 30 22:51:28.987517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:28.987461    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7"} err="failed to get container status \"79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7\": rpc error: code = NotFound desc = could not find container \"79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7\": container with ID starting with 79c5671497e3821c394f9f8a6bd71ab39cf7d1e3c3554b7df1f2929722ba06d7 not found: ID does not exist"
Oct 30 22:51:29.041747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.041715    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:29.098065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.098031    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"]
Oct 30 22:51:29.098556 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.098544    5503 cpu_manager.go:414] "containerMap: removing container" podUID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" containerName="payload-processing"
Oct 30 22:51:29.098613 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.098557    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" containerName="payload-processing"
Oct 30 22:51:29.098693 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.098649    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" containerName="payload-processing"
Oct 30 22:51:29.103241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.103225    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.144862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.144789    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"]
Oct 30 22:51:29.149383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.149358    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-67fbfkr76"]
Oct 30 22:51:29.158098 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.158069    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"]
Oct 30 22:51:29.185009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.184930    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.185009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.184968    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.185188 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.185153    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh546\" (UniqueName: \"kubernetes.io/projected/405a1f97-068c-4506-b99a-d8a905f64cc7-kube-api-access-fh546\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.192817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.192783    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:29.192817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.192817    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:51:29.192969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.192838    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:29.192969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.192861    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:29.286336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.286299    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh546\" (UniqueName: \"kubernetes.io/projected/405a1f97-068c-4506-b99a-d8a905f64cc7-kube-api-access-fh546\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.286484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.286421    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.286484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.286445    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.287215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.287185    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.288848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.288828    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.348583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.348549    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh546\" (UniqueName: \"kubernetes.io/projected/405a1f97-068c-4506-b99a-d8a905f64cc7-kube-api-access-fh546\") pod \"maas-api-revoke-sub-daf0bc5d-p86dc\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.417002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.416975    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:29.785922 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:51:29.785893    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405a1f97_068c_4506_b99a_d8a905f64cc7.slice/crio-65f659f86eeb67ff0fb2005f9ae94b992affb9f65408e18a252d9aa9c8f7f14d WatchSource:0}: Error finding container 65f659f86eeb67ff0fb2005f9ae94b992affb9f65408e18a252d9aa9c8f7f14d: Status 404 returned error can't find the container with id 65f659f86eeb67ff0fb2005f9ae94b992affb9f65408e18a252d9aa9c8f7f14d
Oct 30 22:51:29.790797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.790727    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2a81d9-4a8a-47a1-834d-c6e0abd15785" path="/var/lib/kubelet/pods/6e2a81d9-4a8a-47a1-834d-c6e0abd15785/volumes"
Oct 30 22:51:29.791027 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.791009    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"]
Oct 30 22:51:29.984470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:29.984429    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc" event={"ID":"405a1f97-068c-4506-b99a-d8a905f64cc7","Type":"ContainerStarted","Data":"65f659f86eeb67ff0fb2005f9ae94b992affb9f65408e18a252d9aa9c8f7f14d"}
Oct 30 22:51:30.221568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:30.221534    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:30.994564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:30.994483    5503 generic.go:356] "Generic (PLEG): container finished" podID="405a1f97-068c-4506-b99a-d8a905f64cc7" containerID="00eadb3b057e6665b1cef88a1e8c2a932eae337d71b01c64a5c1afa898ae6614" exitCode=0
Oct 30 22:51:30.994965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:30.994568    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc" event={"ID":"405a1f97-068c-4506-b99a-d8a905f64cc7","Type":"ContainerDied","Data":"00eadb3b057e6665b1cef88a1e8c2a932eae337d71b01c64a5c1afa898ae6614"}
Oct 30 22:51:31.645266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:31.645229    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:32.148152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.148126    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:32.220710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.220670    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-openshift-ca\") pod \"405a1f97-068c-4506-b99a-d8a905f64cc7\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") "
Oct 30 22:51:32.220918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.220732    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405a1f97-068c-4506-b99a-d8a905f64cc7-kube-api-access-fh546\" (UniqueName: \"kubernetes.io/projected/405a1f97-068c-4506-b99a-d8a905f64cc7-kube-api-access-fh546\") pod \"405a1f97-068c-4506-b99a-d8a905f64cc7\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") "
Oct 30 22:51:32.220918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.220876    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-xks-ca\") pod \"405a1f97-068c-4506-b99a-d8a905f64cc7\" (UID: \"405a1f97-068c-4506-b99a-d8a905f64cc7\") "
Oct 30 22:51:32.221057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.220993    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-openshift-ca" pod "405a1f97-068c-4506-b99a-d8a905f64cc7" (UID: "405a1f97-068c-4506-b99a-d8a905f64cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:32.221283 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.221264    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:32.222915 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.222895    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405a1f97-068c-4506-b99a-d8a905f64cc7-kube-api-access-fh546" pod "405a1f97-068c-4506-b99a-d8a905f64cc7" (UID: "405a1f97-068c-4506-b99a-d8a905f64cc7"). InnerVolumeSpecName "kube-api-access-fh546". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:32.223043 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.223025    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-xks-ca" pod "405a1f97-068c-4506-b99a-d8a905f64cc7" (UID: "405a1f97-068c-4506-b99a-d8a905f64cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:32.322631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.322552    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405a1f97-068c-4506-b99a-d8a905f64cc7-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:32.322631 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.322583    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh546\" (UniqueName: \"kubernetes.io/projected/405a1f97-068c-4506-b99a-d8a905f64cc7-kube-api-access-fh546\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:32.885612 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:32.885580    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:33.007834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:33.007807    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"
Oct 30 22:51:33.007834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:33.007823    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc" event={"ID":"405a1f97-068c-4506-b99a-d8a905f64cc7","Type":"ContainerDied","Data":"65f659f86eeb67ff0fb2005f9ae94b992affb9f65408e18a252d9aa9c8f7f14d"}
Oct 30 22:51:33.008054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:33.007865    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f659f86eeb67ff0fb2005f9ae94b992affb9f65408e18a252d9aa9c8f7f14d"
Oct 30 22:51:34.199151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:34.199119    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:34.954704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:34.954670    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:35.724680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:35.724647    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:35.995040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:35.994950    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:36.148714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:36.148663    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:36.259173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:36.259068    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"]
Oct 30 22:51:36.309620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:36.309589    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-f4e241-8bvnn"]
Oct 30 22:51:36.667843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:36.667808    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:37.789273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:37.789238    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12414c1-8181-4930-8e3e-c156cd78fbd2" path="/var/lib/kubelet/pods/f12414c1-8181-4930-8e3e-c156cd78fbd2/volumes"
Oct 30 22:51:40.693562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.693530    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"]
Oct 30 22:51:40.694055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.694040    5503 cpu_manager.go:414] "containerMap: removing container" podUID="405a1f97-068c-4506-b99a-d8a905f64cc7" containerName="revoke-keys"
Oct 30 22:51:40.694100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.694056    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405a1f97-068c-4506-b99a-d8a905f64cc7" containerName="revoke-keys"
Oct 30 22:51:40.694169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.694160    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="405a1f97-068c-4506-b99a-d8a905f64cc7" containerName="revoke-keys"
Oct 30 22:51:40.697445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.697429    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.752959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.752929    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"]
Oct 30 22:51:40.801257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.801225    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:40.801257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.801241    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:40.801257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.801254    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:40.801575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.801560    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:51:40.809022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.808999    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4p7z\" (UniqueName: \"kubernetes.io/projected/8939bcab-c8de-4e8e-b6d7-0035be050fb6-kube-api-access-s4p7z\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.809129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.809059    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.809214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.809149    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.857051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.857014    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:40.910456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.910419    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.910704 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.910593    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4p7z\" (UniqueName: \"kubernetes.io/projected/8939bcab-c8de-4e8e-b6d7-0035be050fb6-kube-api-access-s4p7z\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.910784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.910721    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.911196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.911173    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.913169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.913148    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:40.973393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:40.973329    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4p7z\" (UniqueName: \"kubernetes.io/projected/8939bcab-c8de-4e8e-b6d7-0035be050fb6-kube-api-access-s4p7z\") pod \"maas-api-revoke-sub-cb029b16-7cdjk\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:41.010435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:41.010408    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:41.381705 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:51:41.381672    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8939bcab_c8de_4e8e_b6d7_0035be050fb6.slice/crio-51eff68b1ed6bd441014d6ab6ea789e9e5463de6e19dc9b3277df9c877012b0a WatchSource:0}: Error finding container 51eff68b1ed6bd441014d6ab6ea789e9e5463de6e19dc9b3277df9c877012b0a: Status 404 returned error can't find the container with id 51eff68b1ed6bd441014d6ab6ea789e9e5463de6e19dc9b3277df9c877012b0a
Oct 30 22:51:41.383718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:41.383692    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"]
Oct 30 22:51:42.070563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:42.070477    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk" event={"ID":"8939bcab-c8de-4e8e-b6d7-0035be050fb6","Type":"ContainerStarted","Data":"854dc601df678fa31ebbf716a910da237f162a4a6b9192ecaef2a96f59898142"}
Oct 30 22:51:42.070965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:42.070607    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk" event={"ID":"8939bcab-c8de-4e8e-b6d7-0035be050fb6","Type":"ContainerStarted","Data":"51eff68b1ed6bd441014d6ab6ea789e9e5463de6e19dc9b3277df9c877012b0a"}
Oct 30 22:51:42.192156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:42.192099    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk" podStartSLOduration=2.19208606 podStartE2EDuration="2.19208606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.190956288 +0000 UTC m=+3878.525533057" watchObservedRunningTime="2026-10-30 22:51:42.19208606 +0000 UTC m=+3878.526662829"
Oct 30 22:51:43.078984 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:43.078950    5503 generic.go:356] "Generic (PLEG): container finished" podID="8939bcab-c8de-4e8e-b6d7-0035be050fb6" containerID="854dc601df678fa31ebbf716a910da237f162a4a6b9192ecaef2a96f59898142" exitCode=0
Oct 30 22:51:43.079417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:43.079036    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk" event={"ID":"8939bcab-c8de-4e8e-b6d7-0035be050fb6","Type":"ContainerDied","Data":"854dc601df678fa31ebbf716a910da237f162a4a6b9192ecaef2a96f59898142"}
Oct 30 22:51:43.190201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:43.190160    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:43.348159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:43.348085    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:43.540625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:43.540593    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:44.231366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.231333    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:44.348131 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.348042    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-openshift-ca\") pod \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") "
Oct 30 22:51:44.348311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.348173    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8939bcab-c8de-4e8e-b6d7-0035be050fb6-kube-api-access-s4p7z\" (UniqueName: \"kubernetes.io/projected/8939bcab-c8de-4e8e-b6d7-0035be050fb6-kube-api-access-s4p7z\") pod \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") "
Oct 30 22:51:44.348311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.348241    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-xks-ca\") pod \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\" (UID: \"8939bcab-c8de-4e8e-b6d7-0035be050fb6\") "
Oct 30 22:51:44.348523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.348493    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-openshift-ca" pod "8939bcab-c8de-4e8e-b6d7-0035be050fb6" (UID: "8939bcab-c8de-4e8e-b6d7-0035be050fb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:44.348689 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.348663    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:44.351993 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.351962    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8939bcab-c8de-4e8e-b6d7-0035be050fb6-kube-api-access-s4p7z" pod "8939bcab-c8de-4e8e-b6d7-0035be050fb6" (UID: "8939bcab-c8de-4e8e-b6d7-0035be050fb6"). InnerVolumeSpecName "kube-api-access-s4p7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:44.353927 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.353897    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-xks-ca" pod "8939bcab-c8de-4e8e-b6d7-0035be050fb6" (UID: "8939bcab-c8de-4e8e-b6d7-0035be050fb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:44.450346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.450310    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8939bcab-c8de-4e8e-b6d7-0035be050fb6-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:44.450346 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.450340    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4p7z\" (UniqueName: \"kubernetes.io/projected/8939bcab-c8de-4e8e-b6d7-0035be050fb6-kube-api-access-s4p7z\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:44.506079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.506047    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"]
Oct 30 22:51:44.506577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.506562    5503 cpu_manager.go:414] "containerMap: removing container" podUID="8939bcab-c8de-4e8e-b6d7-0035be050fb6" containerName="revoke-keys"
Oct 30 22:51:44.506680 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.506578    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8939bcab-c8de-4e8e-b6d7-0035be050fb6" containerName="revoke-keys"
Oct 30 22:51:44.506751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.506716    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8939bcab-c8de-4e8e-b6d7-0035be050fb6" containerName="revoke-keys"
Oct 30 22:51:44.510065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.510044    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.564120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.564092    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"]
Oct 30 22:51:44.651765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.651730    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.651943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.651884    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88wlk\" (UniqueName: \"kubernetes.io/projected/bf157e47-ec53-4268-b35d-c7f9ed583625-kube-api-access-88wlk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.652050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.652032    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.752845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.752809    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.753023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.752881    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wlk\" (UniqueName: \"kubernetes.io/projected/bf157e47-ec53-4268-b35d-c7f9ed583625-kube-api-access-88wlk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.753023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.752942    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.753510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.753483    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.755170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.755147    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.817523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.817493    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88wlk\" (UniqueName: \"kubernetes.io/projected/bf157e47-ec53-4268-b35d-c7f9ed583625-kube-api-access-88wlk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.823320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.823298    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:44.953662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:44.953559    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:45.092890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:45.092864    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"
Oct 30 22:51:45.093055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:45.092887    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk" event={"ID":"8939bcab-c8de-4e8e-b6d7-0035be050fb6","Type":"ContainerDied","Data":"51eff68b1ed6bd441014d6ab6ea789e9e5463de6e19dc9b3277df9c877012b0a"}
Oct 30 22:51:45.093055 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:45.092932    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51eff68b1ed6bd441014d6ab6ea789e9e5463de6e19dc9b3277df9c877012b0a"
Oct 30 22:51:45.193850 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:51:45.193820    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf157e47_ec53_4268_b35d_c7f9ed583625.slice/crio-cd86ace428cccba5b77642e4760bdbe294362e5acb474e74b807d14c63f618c3 WatchSource:0}: Error finding container cd86ace428cccba5b77642e4760bdbe294362e5acb474e74b807d14c63f618c3: Status 404 returned error can't find the container with id cd86ace428cccba5b77642e4760bdbe294362e5acb474e74b807d14c63f618c3
Oct 30 22:51:45.197007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:45.196986    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"]
Oct 30 22:51:45.655677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:45.655617    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:46.101719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:46.101693    5503 generic.go:356] "Generic (PLEG): container finished" podID="bf157e47-ec53-4268-b35d-c7f9ed583625" containerID="9a30d7ba69df4946c7b6947974f18d25d590238ba773c4191f1b4566a30de6fc" exitCode=0
Oct 30 22:51:46.101844 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:46.101783    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf" event={"ID":"bf157e47-ec53-4268-b35d-c7f9ed583625","Type":"ContainerDied","Data":"9a30d7ba69df4946c7b6947974f18d25d590238ba773c4191f1b4566a30de6fc"}
Oct 30 22:51:46.101907 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:46.101840    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf" event={"ID":"bf157e47-ec53-4268-b35d-c7f9ed583625","Type":"ContainerStarted","Data":"cd86ace428cccba5b77642e4760bdbe294362e5acb474e74b807d14c63f618c3"}
Oct 30 22:51:47.271108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.271076    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:47.386969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.386878    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf157e47-ec53-4268-b35d-c7f9ed583625-kube-api-access-88wlk\" (UniqueName: \"kubernetes.io/projected/bf157e47-ec53-4268-b35d-c7f9ed583625-kube-api-access-88wlk\") pod \"bf157e47-ec53-4268-b35d-c7f9ed583625\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") "
Oct 30 22:51:47.387149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.386972    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-openshift-ca\") pod \"bf157e47-ec53-4268-b35d-c7f9ed583625\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") "
Oct 30 22:51:47.387149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.387035    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-xks-ca\") pod \"bf157e47-ec53-4268-b35d-c7f9ed583625\" (UID: \"bf157e47-ec53-4268-b35d-c7f9ed583625\") "
Oct 30 22:51:47.387593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.387560    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-openshift-ca" pod "bf157e47-ec53-4268-b35d-c7f9ed583625" (UID: "bf157e47-ec53-4268-b35d-c7f9ed583625"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:47.389365 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.389328    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf157e47-ec53-4268-b35d-c7f9ed583625-kube-api-access-88wlk" pod "bf157e47-ec53-4268-b35d-c7f9ed583625" (UID: "bf157e47-ec53-4268-b35d-c7f9ed583625"). InnerVolumeSpecName "kube-api-access-88wlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:47.389764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.389733    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-xks-ca" pod "bf157e47-ec53-4268-b35d-c7f9ed583625" (UID: "bf157e47-ec53-4268-b35d-c7f9ed583625"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:47.488906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.488867    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:47.488906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.488900    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88wlk\" (UniqueName: \"kubernetes.io/projected/bf157e47-ec53-4268-b35d-c7f9ed583625-kube-api-access-88wlk\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:47.488906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:47.488912    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf157e47-ec53-4268-b35d-c7f9ed583625-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:48.115350 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:48.115280    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"
Oct 30 22:51:48.115508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:48.115279    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf" event={"ID":"bf157e47-ec53-4268-b35d-c7f9ed583625","Type":"ContainerDied","Data":"cd86ace428cccba5b77642e4760bdbe294362e5acb474e74b807d14c63f618c3"}
Oct 30 22:51:48.115508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:48.115408    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd86ace428cccba5b77642e4760bdbe294362e5acb474e74b807d14c63f618c3"
Oct 30 22:51:50.388666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:50.388619    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:54.954117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:54.954074    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:54.957595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:54.957545    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:56.086535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:56.086500    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"]
Oct 30 22:51:56.462564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:56.462531    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:56.616932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:56.616902    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:58.101345 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:58.101305    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:51:58.539551 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:58.539515    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:58.707832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:58.707796    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:59.357371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.357341    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"]
Oct 30 22:51:59.359022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.358982    5503 cpu_manager.go:414] "containerMap: removing container" podUID="bf157e47-ec53-4268-b35d-c7f9ed583625" containerName="revoke-keys"
Oct 30 22:51:59.359022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.359020    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf157e47-ec53-4268-b35d-c7f9ed583625" containerName="revoke-keys"
Oct 30 22:51:59.359326 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.359302    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf157e47-ec53-4268-b35d-c7f9ed583625" containerName="revoke-keys"
Oct 30 22:51:59.364240 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.364217    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.416201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.416168    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"]
Oct 30 22:51:59.423662 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.423611    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.423820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.423736    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sfdx\" (UniqueName: \"kubernetes.io/projected/56d45169-b3f3-41b1-81a8-8dc4a0455942-kube-api-access-8sfdx\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.423820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.423790    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.464151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.464124    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:59.464401 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.464386    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:51:59.464461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.464414    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:51:59.468323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.468305    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:51:59.524801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.524776    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.524903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.524808    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sfdx\" (UniqueName: \"kubernetes.io/projected/56d45169-b3f3-41b1-81a8-8dc4a0455942-kube-api-access-8sfdx\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.524903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.524832    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.525568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.525546    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.527212 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.527180    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.527545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.527513    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:51:59.587375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.587340    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sfdx\" (UniqueName: \"kubernetes.io/projected/56d45169-b3f3-41b1-81a8-8dc4a0455942-kube-api-access-8sfdx\") pod \"maas-api-revoke-sub-2faed267-9m62d\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.678509 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:51:59.678482    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:51:59.829019 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:59.828831    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-12328dd500c5: secret "agc-cred-12328dd500c5" not found
Oct 30 22:51:59.829019 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:59.828862    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-d4b0ee4a268a: secret "agc-cred-d4b0ee4a268a" not found
Oct 30 22:51:59.829019 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:59.828880    5503 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm: [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:51:59.829019 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:51:59.828962    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials podName:b0fc80fd-a0d8-49c4-a724-83ae164b54d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:52:00.328939723 +0000 UTC m=+3896.663516490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9") : [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:00.058991 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:52:00.058962    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d45169_b3f3_41b1_81a8_8dc4a0455942.slice/crio-a794f8b87d3d922ad4ffc61dc2b9f41e43a02e587aace5caef015c30a39b2b44 WatchSource:0}: Error finding container a794f8b87d3d922ad4ffc61dc2b9f41e43a02e587aace5caef015c30a39b2b44: Status 404 returned error can't find the container with id a794f8b87d3d922ad4ffc61dc2b9f41e43a02e587aace5caef015c30a39b2b44
Oct 30 22:52:00.061517 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:00.061493    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"]
Oct 30 22:52:00.190865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:00.190824    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d" event={"ID":"56d45169-b3f3-41b1-81a8-8dc4a0455942","Type":"ContainerStarted","Data":"a794f8b87d3d922ad4ffc61dc2b9f41e43a02e587aace5caef015c30a39b2b44"}
Oct 30 22:52:00.333964 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:00.333938    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-12328dd500c5: secret "agc-cred-12328dd500c5" not found
Oct 30 22:52:00.333964 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:00.333967    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-d4b0ee4a268a: secret "agc-cred-d4b0ee4a268a" not found
Oct 30 22:52:00.334149 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:00.333985    5503 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm: [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:00.334149 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:00.334035    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials podName:b0fc80fd-a0d8-49c4-a724-83ae164b54d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:52:01.334022159 +0000 UTC m=+3897.668598910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9") : [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:00.348788 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:00.348764    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:52:01.086978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:01.086910    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" podUID="dcb257f1-291e-4efa-a34f-27c8df655ea6" containerName="payload-processing" containerID="cri-o://3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85" gracePeriod=25
Oct 30 22:52:01.198753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:01.198716    5503 generic.go:356] "Generic (PLEG): container finished" podID="56d45169-b3f3-41b1-81a8-8dc4a0455942" containerID="a5d79c7e61420ac2abe4e9b86e73683125233ae491ec40842bc84103118117ea" exitCode=0
Oct 30 22:52:01.198930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:01.198801    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d" event={"ID":"56d45169-b3f3-41b1-81a8-8dc4a0455942","Type":"ContainerDied","Data":"a5d79c7e61420ac2abe4e9b86e73683125233ae491ec40842bc84103118117ea"}
Oct 30 22:52:01.344988 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:01.344957    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-12328dd500c5: secret "agc-cred-12328dd500c5" not found
Oct 30 22:52:01.344988 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:01.344985    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-d4b0ee4a268a: secret "agc-cred-d4b0ee4a268a" not found
Oct 30 22:52:01.344988 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:01.344998    5503 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm: [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:01.345241 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:01.345059    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials podName:b0fc80fd-a0d8-49c4-a724-83ae164b54d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:52:03.345041318 +0000 UTC m=+3899.679618079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9") : [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:02.343802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.343775    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:52:02.458685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.458564    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-xks-ca\") pod \"56d45169-b3f3-41b1-81a8-8dc4a0455942\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") "
Oct 30 22:52:02.458872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.458704    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d45169-b3f3-41b1-81a8-8dc4a0455942-kube-api-access-8sfdx\" (UniqueName: \"kubernetes.io/projected/56d45169-b3f3-41b1-81a8-8dc4a0455942-kube-api-access-8sfdx\") pod \"56d45169-b3f3-41b1-81a8-8dc4a0455942\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") "
Oct 30 22:52:02.458872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.458826    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-openshift-ca\") pod \"56d45169-b3f3-41b1-81a8-8dc4a0455942\" (UID: \"56d45169-b3f3-41b1-81a8-8dc4a0455942\") "
Oct 30 22:52:02.459621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.459586    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-openshift-ca" pod "56d45169-b3f3-41b1-81a8-8dc4a0455942" (UID: "56d45169-b3f3-41b1-81a8-8dc4a0455942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:02.461311 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.461274    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d45169-b3f3-41b1-81a8-8dc4a0455942-kube-api-access-8sfdx" pod "56d45169-b3f3-41b1-81a8-8dc4a0455942" (UID: "56d45169-b3f3-41b1-81a8-8dc4a0455942"). InnerVolumeSpecName "kube-api-access-8sfdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:02.461449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.461342    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-xks-ca" pod "56d45169-b3f3-41b1-81a8-8dc4a0455942" (UID: "56d45169-b3f3-41b1-81a8-8dc4a0455942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:02.560348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.560312    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:02.560348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.560345    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sfdx\" (UniqueName: \"kubernetes.io/projected/56d45169-b3f3-41b1-81a8-8dc4a0455942-kube-api-access-8sfdx\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:02.560575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:02.560359    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d45169-b3f3-41b1-81a8-8dc4a0455942-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.213488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:03.213455    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"
Oct 30 22:52:03.213488 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:03.213472    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d" event={"ID":"56d45169-b3f3-41b1-81a8-8dc4a0455942","Type":"ContainerDied","Data":"a794f8b87d3d922ad4ffc61dc2b9f41e43a02e587aace5caef015c30a39b2b44"}
Oct 30 22:52:03.213736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:03.213531    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a794f8b87d3d922ad4ffc61dc2b9f41e43a02e587aace5caef015c30a39b2b44"
Oct 30 22:52:03.369962 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:03.369930    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-12328dd500c5: secret "agc-cred-12328dd500c5" not found
Oct 30 22:52:03.369962 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:03.369956    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-d4b0ee4a268a: secret "agc-cred-d4b0ee4a268a" not found
Oct 30 22:52:03.369962 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:03.369970    5503 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm: [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:03.370422 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:03.370026    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials podName:b0fc80fd-a0d8-49c4-a724-83ae164b54d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:52:07.370010435 +0000 UTC m=+3903.704587200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9") : [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:04.143941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:04.143889    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:04.428257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:04.428233    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:52:04.440161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:04.440141    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:52:04.953872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:04.953834    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:07.424357 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:07.424323    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-12328dd500c5: secret "agc-cred-12328dd500c5" not found
Oct 30 22:52:07.424357 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:07.424352    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-d4b0ee4a268a: secret "agc-cred-d4b0ee4a268a" not found
Oct 30 22:52:07.424357 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:07.424365    5503 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm: [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:07.424921 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:07.424433    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials podName:b0fc80fd-a0d8-49c4-a724-83ae164b54d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:52:15.424418023 +0000 UTC m=+3911.758994769 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9") : [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found]
Oct 30 22:52:08.198514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:08.198456    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:09.278363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:09.278329    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:09.622737 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:09.622630    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:09.894944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:09.894847    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:10.097197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:10.097147    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"]
Oct 30 22:52:10.146582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:10.146510    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f39eb7-pgzvf"]
Oct 30 22:52:10.271505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:10.271473    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" podUID="dcb257f1-291e-4efa-a34f-27c8df655ea6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:10.555443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:10.555414    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:10.858862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:10.858770    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:11.130831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.130753    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:11.287371 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.287342    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:11.370223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.370187    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"]
Oct 30 22:52:11.420668 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.420629    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-e65019-9d5jp"]
Oct 30 22:52:11.791499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.791413    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5c6c8f-f522-41a0-8970-5166b4d6bb02" path="/var/lib/kubelet/pods/7f5c6c8f-f522-41a0-8970-5166b4d6bb02/volumes"
Oct 30 22:52:11.791935 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.791917    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf157e47-ec53-4268-b35d-c7f9ed583625" path="/var/lib/kubelet/pods/bf157e47-ec53-4268-b35d-c7f9ed583625/volumes"
Oct 30 22:52:11.796246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.796217    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:11.814093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.814063    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"]
Oct 30 22:52:11.815413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.815375    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:11.818904 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.818883    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"]
Oct 30 22:52:11.958342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:11.958293    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:12.118428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:12.118336    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:12.605802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:12.605765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:15.520548 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:15.520520    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-12328dd500c5: secret "agc-cred-12328dd500c5" not found
Oct 30 22:52:15.520548 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:15.520542    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-d4b0ee4a268a: secret "agc-cred-d4b0ee4a268a" not found
Oct 30 22:52:15.520548 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:15.520549    5503 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-1b5075/agc-cred-f71a8cda136e: secret "agc-cred-f71a8cda136e" not found
Oct 30 22:52:15.520548 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:15.520558    5503 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm: [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found, secret "agc-cred-f71a8cda136e" not found]
Oct 30 22:52:15.521072 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:15.520619    5503 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials podName:b0fc80fd-a0d8-49c4-a724-83ae164b54d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:52:31.520604595 +0000 UTC m=+3927.855181355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9") : [secret "agc-cred-12328dd500c5" not found, secret "agc-cred-d4b0ee4a268a" not found, secret "agc-cred-f71a8cda136e" not found]
Oct 30 22:52:16.018197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:16.018161    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:16.815692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:16.815653    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing" containerID="cri-o://58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2" gracePeriod=25
Oct 30 22:52:16.819576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:16.819474    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" podUID="b0fc80fd-a0d8-49c4-a724-83ae164b54d9" containerName="payload-processing" containerID="cri-o://49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe" gracePeriod=25
Oct 30 22:52:17.094962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.094939    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:52:17.102666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.102625    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:52:17.242249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242217    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-routing-overlay-volume\") pod \"fc35d222-4674-4ffd-8373-de52d20deb4e\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") "
Oct 30 22:52:17.242471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242265    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-plugins-config-volume\") pod \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") "
Oct 30 22:52:17.242471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242309    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-provider-credentials\") pod \"fc35d222-4674-4ffd-8373-de52d20deb4e\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") "
Oct 30 22:52:17.242471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242377    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-routing-overlay-volume\") pod \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") "
Oct 30 22:52:17.242471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242405    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-kube-api-access-wdqq5\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-kube-api-access-wdqq5\") pod \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") "
Oct 30 22:52:17.242471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242452    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-kube-api-access-zgvkw\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-kube-api-access-zgvkw\") pod \"fc35d222-4674-4ffd-8373-de52d20deb4e\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") "
Oct 30 22:52:17.242777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242586    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-plugins-config-volume\") pod \"fc35d222-4674-4ffd-8373-de52d20deb4e\" (UID: \"fc35d222-4674-4ffd-8373-de52d20deb4e\") "
Oct 30 22:52:17.242777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242687    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials\") pod \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\" (UID: \"b0fc80fd-a0d8-49c4-a724-83ae164b54d9\") "
Oct 30 22:52:17.242777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242704    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-routing-overlay-volume" pod "fc35d222-4674-4ffd-8373-de52d20deb4e" (UID: "fc35d222-4674-4ffd-8373-de52d20deb4e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:17.242777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242758    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-plugins-config-volume" pod "b0fc80fd-a0d8-49c4-a724-83ae164b54d9" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:17.242992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.242795    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-routing-overlay-volume" pod "b0fc80fd-a0d8-49c4-a724-83ae164b54d9" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:17.243352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.243329    5503 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-routing-overlay-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.244011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.243355    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.244011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.243362    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-plugins-config-volume" pod "fc35d222-4674-4ffd-8373-de52d20deb4e" (UID: "fc35d222-4674-4ffd-8373-de52d20deb4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:17.244011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.243372    5503 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-routing-overlay-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.245499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.245203    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:17.246082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.246057    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials" pod "b0fc80fd-a0d8-49c4-a724-83ae164b54d9" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:17.246512 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.246489    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-provider-credentials" pod "fc35d222-4674-4ffd-8373-de52d20deb4e" (UID: "fc35d222-4674-4ffd-8373-de52d20deb4e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:17.246884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.246864    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-kube-api-access-zgvkw" pod "fc35d222-4674-4ffd-8373-de52d20deb4e" (UID: "fc35d222-4674-4ffd-8373-de52d20deb4e"). InnerVolumeSpecName "kube-api-access-zgvkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:17.247804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.247780    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-kube-api-access-wdqq5" pod "b0fc80fd-a0d8-49c4-a724-83ae164b54d9" (UID: "b0fc80fd-a0d8-49c4-a724-83ae164b54d9"). InnerVolumeSpecName "kube-api-access-wdqq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:17.306890 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.306858    5503 generic.go:356] "Generic (PLEG): container finished" podID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerID="58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2" exitCode=0
Oct 30 22:52:17.307093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.306909    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"
Oct 30 22:52:17.307093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.306946    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" event={"ID":"fc35d222-4674-4ffd-8373-de52d20deb4e","Type":"ContainerDied","Data":"58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2"}
Oct 30 22:52:17.307093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.307009    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22" event={"ID":"fc35d222-4674-4ffd-8373-de52d20deb4e","Type":"ContainerDied","Data":"478dc7e5c2098b69602fca7e47e008f63f5074ba44cb644ceb6856fb0a824034"}
Oct 30 22:52:17.307093 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.307038    5503 scope.go:122] "RemoveContainer" containerID="58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2"
Oct 30 22:52:17.308519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.308492    5503 generic.go:356] "Generic (PLEG): container finished" podID="b0fc80fd-a0d8-49c4-a724-83ae164b54d9" containerID="49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe" exitCode=0
Oct 30 22:52:17.308684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.308532    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"
Oct 30 22:52:17.308684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.308568    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" event={"ID":"b0fc80fd-a0d8-49c4-a724-83ae164b54d9","Type":"ContainerDied","Data":"49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe"}
Oct 30 22:52:17.308684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.308610    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm" event={"ID":"b0fc80fd-a0d8-49c4-a724-83ae164b54d9","Type":"ContainerDied","Data":"c560b5d0c62dcf58ccea0bf5e04cc4df31c47683389833378359968fcaf5efb8"}
Oct 30 22:52:17.319376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.319151    5503 scope.go:122] "RemoveContainer" containerID="58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2"
Oct 30 22:52:17.327966 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:17.327939    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2\": container with ID starting with 58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2 not found: ID does not exist" containerID="58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2"
Oct 30 22:52:17.328091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.327973    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2"} err="failed to get container status \"58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2\": rpc error: code = NotFound desc = could not find container \"58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2\": container with ID starting with 58fc54acec87487b5708e3842245872a973c4fca4591da5358ba9834183c9cd2 not found: ID does not exist"
Oct 30 22:52:17.328091 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.327993    5503 scope.go:122] "RemoveContainer" containerID="49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe"
Oct 30 22:52:17.338222 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.338192    5503 scope.go:122] "RemoveContainer" containerID="49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe"
Oct 30 22:52:17.338474 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:17.338455    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe\": container with ID starting with 49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe not found: ID does not exist" containerID="49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe"
Oct 30 22:52:17.338539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.338482    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe"} err="failed to get container status \"49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe\": rpc error: code = NotFound desc = could not find container \"49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe\": container with ID starting with 49b7662b974a17252aa474109ff917d643d0072f276555af7e39c4a7e70818fe not found: ID does not exist"
Oct 30 22:52:17.344684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.344664    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdqq5\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-kube-api-access-wdqq5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.344684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.344685    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgvkw\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-kube-api-access-zgvkw\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.344814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.344695    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc35d222-4674-4ffd-8373-de52d20deb4e-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.344814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.344704    5503 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0fc80fd-a0d8-49c4-a724-83ae164b54d9-provider-credentials\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.344814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.344712    5503 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc35d222-4674-4ffd-8373-de52d20deb4e-provider-credentials\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:17.435898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.435860    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:17.479618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.479582    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"]
Oct 30 22:52:17.492221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.492190    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-7678xrx22"]
Oct 30 22:52:17.643717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.637613    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"]
Oct 30 22:52:17.643717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.641315    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1b5075/payload-processing-external-model-e2e-extmod-b-1b5075-6b4fw4kgm"]
Oct 30 22:52:17.654655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.647132    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:17.789241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.789206    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fc80fd-a0d8-49c4-a724-83ae164b54d9" path="/var/lib/kubelet/pods/b0fc80fd-a0d8-49c4-a724-83ae164b54d9/volumes"
Oct 30 22:52:17.789655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:17.789622    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" path="/var/lib/kubelet/pods/fc35d222-4674-4ffd-8373-de52d20deb4e/volumes"
Oct 30 22:52:18.324779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.324741    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"]
Oct 30 22:52:18.325263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325249    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b0fc80fd-a0d8-49c4-a724-83ae164b54d9" containerName="payload-processing"
Oct 30 22:52:18.325324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325265    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fc80fd-a0d8-49c4-a724-83ae164b54d9" containerName="payload-processing"
Oct 30 22:52:18.325324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325277    5503 cpu_manager.go:414] "containerMap: removing container" podUID="56d45169-b3f3-41b1-81a8-8dc4a0455942" containerName="revoke-keys"
Oct 30 22:52:18.325324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325288    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d45169-b3f3-41b1-81a8-8dc4a0455942" containerName="revoke-keys"
Oct 30 22:52:18.325324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325313    5503 cpu_manager.go:414] "containerMap: removing container" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing"
Oct 30 22:52:18.325324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325318    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing"
Oct 30 22:52:18.325483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325426    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fc80fd-a0d8-49c4-a724-83ae164b54d9" containerName="payload-processing"
Oct 30 22:52:18.325483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325436    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d45169-b3f3-41b1-81a8-8dc4a0455942" containerName="revoke-keys"
Oct 30 22:52:18.325483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.325445    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc35d222-4674-4ffd-8373-de52d20deb4e" containerName="payload-processing"
Oct 30 22:52:18.330105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.330083    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.383719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.383689    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"]
Oct 30 22:52:18.384241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.384219    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:18.384241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.384221    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:18.400408 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.400377    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:18.400582 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.400517    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:52:18.456946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.456908    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.456946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.456952    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q9dn\" (UniqueName: \"kubernetes.io/projected/02f35053-7342-4f66-bfe2-0190dea22b23-kube-api-access-7q9dn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.457217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.456987    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.557882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.557846    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.558084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.557892    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q9dn\" (UniqueName: \"kubernetes.io/projected/02f35053-7342-4f66-bfe2-0190dea22b23-kube-api-access-7q9dn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.558084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.557962    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.558522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.558495    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.560424 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.560397    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.620297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.620226    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q9dn\" (UniqueName: \"kubernetes.io/projected/02f35053-7342-4f66-bfe2-0190dea22b23-kube-api-access-7q9dn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:18.643213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:18.643185    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:19.020326 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:52:19.020287    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f35053_7342_4f66_bfe2_0190dea22b23.slice/crio-9d64f3ec49832cdcf1d9c1a217eb07613daaec668a77f5a6f008e89814000a9d WatchSource:0}: Error finding container 9d64f3ec49832cdcf1d9c1a217eb07613daaec668a77f5a6f008e89814000a9d: Status 404 returned error can't find the container with id 9d64f3ec49832cdcf1d9c1a217eb07613daaec668a77f5a6f008e89814000a9d
Oct 30 22:52:19.022886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:19.022858    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"]
Oct 30 22:52:19.025464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:19.025442    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.178121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:19.178082    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.326777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:19.326735    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28" event={"ID":"02f35053-7342-4f66-bfe2-0190dea22b23","Type":"ContainerStarted","Data":"9d64f3ec49832cdcf1d9c1a217eb07613daaec668a77f5a6f008e89814000a9d"}
Oct 30 22:52:19.343226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:19.343196    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:19.496557 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:19.495960    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:20.272161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:20.272128    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" podUID="dcb257f1-291e-4efa-a34f-27c8df655ea6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:20.356187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:20.356146    5503 generic.go:356] "Generic (PLEG): container finished" podID="02f35053-7342-4f66-bfe2-0190dea22b23" containerID="fbfdfb693ebd4e5f0be32c1cec5fb05cd729b597b8719980a623d1c30eec56df" exitCode=0
Oct 30 22:52:20.356752 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:20.356289    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28" event={"ID":"02f35053-7342-4f66-bfe2-0190dea22b23","Type":"ContainerDied","Data":"fbfdfb693ebd4e5f0be32c1cec5fb05cd729b597b8719980a623d1c30eec56df"}
Oct 30 22:52:20.778327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:20.778297    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:21.199012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.198980    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:21.355675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.355589    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:52:21.363888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.363860    5503 generic.go:356] "Generic (PLEG): container finished" podID="dcb257f1-291e-4efa-a34f-27c8df655ea6" containerID="3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85" exitCode=0
Oct 30 22:52:21.364217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.363904    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"
Oct 30 22:52:21.364217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.363943    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" event={"ID":"dcb257f1-291e-4efa-a34f-27c8df655ea6","Type":"ContainerDied","Data":"3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85"}
Oct 30 22:52:21.364217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.363995    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz" event={"ID":"dcb257f1-291e-4efa-a34f-27c8df655ea6","Type":"ContainerDied","Data":"eb9d0effeefefc4ea677643285ddc84c043fa1f0c61673364f1787cb62eaf3a3"}
Oct 30 22:52:21.364217 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.364018    5503 scope.go:122] "RemoveContainer" containerID="3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85"
Oct 30 22:52:21.373353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.373333    5503 scope.go:122] "RemoveContainer" containerID="3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85"
Oct 30 22:52:21.373682 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:21.373629    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85\": container with ID starting with 3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85 not found: ID does not exist" containerID="3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85"
Oct 30 22:52:21.373813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.373691    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85"} err="failed to get container status \"3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85\": rpc error: code = NotFound desc = could not find container \"3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85\": container with ID starting with 3e824ae695d0818b112cf0ec08c2d1cf03001862049de74a0829a10eab7a5f85 not found: ID does not exist"
Oct 30 22:52:21.491703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.491678    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb257f1-291e-4efa-a34f-27c8df655ea6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb257f1-291e-4efa-a34f-27c8df655ea6-plugins-config-volume\") pod \"dcb257f1-291e-4efa-a34f-27c8df655ea6\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") "
Oct 30 22:52:21.491831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.491781    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb257f1-291e-4efa-a34f-27c8df655ea6-kube-api-access-8nm29\" (UniqueName: \"kubernetes.io/projected/dcb257f1-291e-4efa-a34f-27c8df655ea6-kube-api-access-8nm29\") pod \"dcb257f1-291e-4efa-a34f-27c8df655ea6\" (UID: \"dcb257f1-291e-4efa-a34f-27c8df655ea6\") "
Oct 30 22:52:21.492092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.492066    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb257f1-291e-4efa-a34f-27c8df655ea6-plugins-config-volume" pod "dcb257f1-291e-4efa-a34f-27c8df655ea6" (UID: "dcb257f1-291e-4efa-a34f-27c8df655ea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:21.492292 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.492269    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb257f1-291e-4efa-a34f-27c8df655ea6-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:21.493879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.493856    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb257f1-291e-4efa-a34f-27c8df655ea6-kube-api-access-8nm29" pod "dcb257f1-291e-4efa-a34f-27c8df655ea6" (UID: "dcb257f1-291e-4efa-a34f-27c8df655ea6"). InnerVolumeSpecName "kube-api-access-8nm29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:21.507781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.507761    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:21.592591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.592564    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-openshift-ca\") pod \"02f35053-7342-4f66-bfe2-0190dea22b23\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") "
Oct 30 22:52:21.592765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.592624    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f35053-7342-4f66-bfe2-0190dea22b23-kube-api-access-7q9dn\" (UniqueName: \"kubernetes.io/projected/02f35053-7342-4f66-bfe2-0190dea22b23-kube-api-access-7q9dn\") pod \"02f35053-7342-4f66-bfe2-0190dea22b23\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") "
Oct 30 22:52:21.592765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.592727    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-xks-ca\") pod \"02f35053-7342-4f66-bfe2-0190dea22b23\" (UID: \"02f35053-7342-4f66-bfe2-0190dea22b23\") "
Oct 30 22:52:21.593028 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.593000    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-openshift-ca" pod "02f35053-7342-4f66-bfe2-0190dea22b23" (UID: "02f35053-7342-4f66-bfe2-0190dea22b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:21.593159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.593137    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:21.593308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.593180    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nm29\" (UniqueName: \"kubernetes.io/projected/dcb257f1-291e-4efa-a34f-27c8df655ea6-kube-api-access-8nm29\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:21.594868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.594841    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-xks-ca" pod "02f35053-7342-4f66-bfe2-0190dea22b23" (UID: "02f35053-7342-4f66-bfe2-0190dea22b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:21.594955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.594861    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f35053-7342-4f66-bfe2-0190dea22b23-kube-api-access-7q9dn" pod "02f35053-7342-4f66-bfe2-0190dea22b23" (UID: "02f35053-7342-4f66-bfe2-0190dea22b23"). InnerVolumeSpecName "kube-api-access-7q9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:21.694296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.694260    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02f35053-7342-4f66-bfe2-0190dea22b23-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:21.694296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.694290    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q9dn\" (UniqueName: \"kubernetes.io/projected/02f35053-7342-4f66-bfe2-0190dea22b23-kube-api-access-7q9dn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:21.790046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.790015    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"]
Oct 30 22:52:21.791697 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:21.791658    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f39eb7-6cdcd8d7b5-nm8kz"]
Oct 30 22:52:22.186807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:22.186772    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:22.370877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:22.370792    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"
Oct 30 22:52:22.370877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:22.370824    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28" event={"ID":"02f35053-7342-4f66-bfe2-0190dea22b23","Type":"ContainerDied","Data":"9d64f3ec49832cdcf1d9c1a217eb07613daaec668a77f5a6f008e89814000a9d"}
Oct 30 22:52:22.371360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:22.370879    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d64f3ec49832cdcf1d9c1a217eb07613daaec668a77f5a6f008e89814000a9d"
Oct 30 22:52:23.077189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:23.077146    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:23.316162 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:23.316127    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:23.789539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:23.789506    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb257f1-291e-4efa-a34f-27c8df655ea6" path="/var/lib/kubelet/pods/dcb257f1-291e-4efa-a34f-27c8df655ea6/volumes"
Oct 30 22:52:25.647117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:25.647082    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:25.874005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:25.873934    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:26.034820 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.034759    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:26.178552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.178515    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"]
Oct 30 22:52:26.179034 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.179018    5503 cpu_manager.go:414] "containerMap: removing container" podUID="dcb257f1-291e-4efa-a34f-27c8df655ea6" containerName="payload-processing"
Oct 30 22:52:26.179089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.179036    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb257f1-291e-4efa-a34f-27c8df655ea6" containerName="payload-processing"
Oct 30 22:52:26.179089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.179060    5503 cpu_manager.go:414] "containerMap: removing container" podUID="02f35053-7342-4f66-bfe2-0190dea22b23" containerName="revoke-keys"
Oct 30 22:52:26.179089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.179065    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f35053-7342-4f66-bfe2-0190dea22b23" containerName="revoke-keys"
Oct 30 22:52:26.179183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.179141    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f35053-7342-4f66-bfe2-0190dea22b23" containerName="revoke-keys"
Oct 30 22:52:26.179183 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.179157    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb257f1-291e-4efa-a34f-27c8df655ea6" containerName="payload-processing"
Oct 30 22:52:26.182645 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.182620    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.237997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.237970    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:26.239466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.239438    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"]
Oct 30 22:52:26.287910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.287839    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:52:26.287910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.287857    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:26.287910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.287879    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:26.345682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.345647    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.345859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.345796    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4499\" (UniqueName: \"kubernetes.io/projected/3c668030-43ca-4cb6-8bbb-b4313761ed81-kube-api-access-c4499\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.345916 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.345861    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.447227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.447187    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4499\" (UniqueName: \"kubernetes.io/projected/3c668030-43ca-4cb6-8bbb-b4313761ed81-kube-api-access-c4499\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.447455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.447361    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.447455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.447437    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.448007 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.447976    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.449807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.449787    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.509990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.509953    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4499\" (UniqueName: \"kubernetes.io/projected/3c668030-43ca-4cb6-8bbb-b4313761ed81-kube-api-access-c4499\") pod \"maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:26.795795 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:26.795763    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:27.166704 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:52:27.166622    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c668030_43ca_4cb6_8bbb_b4313761ed81.slice/crio-e8970ca7f5e4f7db8b8addf3e01d34b332acb23ef892998374ac4071bc3ab7af WatchSource:0}: Error finding container e8970ca7f5e4f7db8b8addf3e01d34b332acb23ef892998374ac4071bc3ab7af: Status 404 returned error can't find the container with id e8970ca7f5e4f7db8b8addf3e01d34b332acb23ef892998374ac4071bc3ab7af
Oct 30 22:52:27.169356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:27.169206    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"]
Oct 30 22:52:27.404593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:27.404558    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf" event={"ID":"3c668030-43ca-4cb6-8bbb-b4313761ed81","Type":"ContainerStarted","Data":"e8970ca7f5e4f7db8b8addf3e01d34b332acb23ef892998374ac4071bc3ab7af"}
Oct 30 22:52:27.862199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:27.862120    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:28.023057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:28.023031    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:28.100446 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:28.100414    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:52:28.178320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:28.178294    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:28.412171 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:28.412105    5503 generic.go:356] "Generic (PLEG): container finished" podID="3c668030-43ca-4cb6-8bbb-b4313761ed81" containerID="c3492554465f6c607cb7c8055c8ff14d26d3981a35ab5f5dde0a7c3e45fc8bed" exitCode=0
Oct 30 22:52:28.412366 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:28.412205    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf" event={"ID":"3c668030-43ca-4cb6-8bbb-b4313761ed81","Type":"ContainerDied","Data":"c3492554465f6c607cb7c8055c8ff14d26d3981a35ab5f5dde0a7c3e45fc8bed"}
Oct 30 22:52:29.562258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.562226    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:29.580257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.580232    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-xks-ca\") pod \"3c668030-43ca-4cb6-8bbb-b4313761ed81\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") "
Oct 30 22:52:29.580360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.580265    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c668030-43ca-4cb6-8bbb-b4313761ed81-kube-api-access-c4499\" (UniqueName: \"kubernetes.io/projected/3c668030-43ca-4cb6-8bbb-b4313761ed81-kube-api-access-c4499\") pod \"3c668030-43ca-4cb6-8bbb-b4313761ed81\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") "
Oct 30 22:52:29.582269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.582240    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-xks-ca" pod "3c668030-43ca-4cb6-8bbb-b4313761ed81" (UID: "3c668030-43ca-4cb6-8bbb-b4313761ed81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:29.582377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.582296    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c668030-43ca-4cb6-8bbb-b4313761ed81-kube-api-access-c4499" pod "3c668030-43ca-4cb6-8bbb-b4313761ed81" (UID: "3c668030-43ca-4cb6-8bbb-b4313761ed81"). InnerVolumeSpecName "kube-api-access-c4499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:29.681087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.680987    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-openshift-ca\") pod \"3c668030-43ca-4cb6-8bbb-b4313761ed81\" (UID: \"3c668030-43ca-4cb6-8bbb-b4313761ed81\") "
Oct 30 22:52:29.681360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.681339    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.681443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.681366    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4499\" (UniqueName: \"kubernetes.io/projected/3c668030-43ca-4cb6-8bbb-b4313761ed81-kube-api-access-c4499\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:29.681443 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.681360    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-openshift-ca" pod "3c668030-43ca-4cb6-8bbb-b4313761ed81" (UID: "3c668030-43ca-4cb6-8bbb-b4313761ed81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:29.782355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:29.782323    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c668030-43ca-4cb6-8bbb-b4313761ed81-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:30.325559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:30.322453    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:30.430246 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:30.430209    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf" event={"ID":"3c668030-43ca-4cb6-8bbb-b4313761ed81","Type":"ContainerDied","Data":"e8970ca7f5e4f7db8b8addf3e01d34b332acb23ef892998374ac4071bc3ab7af"}
Oct 30 22:52:30.430710 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:30.430688    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8970ca7f5e4f7db8b8addf3e01d34b332acb23ef892998374ac4071bc3ab7af"
Oct 30 22:52:30.431060 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:30.430514    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"
Oct 30 22:52:30.560041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:30.560004    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:30.915620 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:30.915589    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:31.997382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:31.997331    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:33.206453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:33.206420    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:33.920553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:33.920516    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:34.111463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.111436    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:34.490011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.489971    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"]
Oct 30 22:52:34.490746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.490722    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3c668030-43ca-4cb6-8bbb-b4313761ed81" containerName="revoke-keys"
Oct 30 22:52:34.490922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.490898    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c668030-43ca-4cb6-8bbb-b4313761ed81" containerName="revoke-keys"
Oct 30 22:52:34.491064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.491053    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c668030-43ca-4cb6-8bbb-b4313761ed81" containerName="revoke-keys"
Oct 30 22:52:34.494747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.494727    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:34.541847 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.541814    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f6646e-5cf5-4da1-9ba3-70b826b59525-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:34.542041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.541976    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfgq\" (UniqueName: \"kubernetes.io/projected/25f6646e-5cf5-4da1-9ba3-70b826b59525-kube-api-access-flfgq\") pod \"payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:34.547429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.547402    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"]
Oct 30 22:52:34.595691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.595628    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3b871a\""
Oct 30 22:52:34.595691 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.595675    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3b871a-dockercfg-qhq5x\""
Oct 30 22:52:34.643204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.643171    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfgq\" (UniqueName: \"kubernetes.io/projected/25f6646e-5cf5-4da1-9ba3-70b826b59525-kube-api-access-flfgq\") pod \"payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:34.643370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.643290    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f6646e-5cf5-4da1-9ba3-70b826b59525-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:34.644010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.643991    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f6646e-5cf5-4da1-9ba3-70b826b59525-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:34.705690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.705655    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfgq\" (UniqueName: \"kubernetes.io/projected/25f6646e-5cf5-4da1-9ba3-70b826b59525-kube-api-access-flfgq\") pod \"payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:34.809606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:34.809528    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:35.183803 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:52:35.183772    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f6646e_5cf5_4da1_9ba3_70b826b59525.slice/crio-01fb504681e9365acaa3d7a454e4c970d63b90e39a1d2742890ac24c751cf8b8 WatchSource:0}: Error finding container 01fb504681e9365acaa3d7a454e4c970d63b90e39a1d2742890ac24c751cf8b8: Status 404 returned error can't find the container with id 01fb504681e9365acaa3d7a454e4c970d63b90e39a1d2742890ac24c751cf8b8
Oct 30 22:52:35.186618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:35.186591    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"]
Oct 30 22:52:35.201473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:35.201443    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:35.486673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:35.486558    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf" event={"ID":"25f6646e-5cf5-4da1-9ba3-70b826b59525","Type":"ContainerStarted","Data":"6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15"}
Oct 30 22:52:35.486673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:35.486617    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf" event={"ID":"25f6646e-5cf5-4da1-9ba3-70b826b59525","Type":"ContainerStarted","Data":"01fb504681e9365acaa3d7a454e4c970d63b90e39a1d2742890ac24c751cf8b8"}
Oct 30 22:52:35.604926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:35.604876    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf" podStartSLOduration=1.604863417 podStartE2EDuration="1.604863417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:35.604239044 +0000 UTC m=+3931.938815812" watchObservedRunningTime="2026-10-30 22:52:35.604863417 +0000 UTC m=+3931.939440183"
Oct 30 22:52:36.492807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:36.492770    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:36.714486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:36.714454    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.022437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:37.022396    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.181280 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:37.181240    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.220806 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:37.220770    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"]
Oct 30 22:52:37.271685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:37.271653    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-293c80-wj8l7"]
Oct 30 22:52:37.672953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:37.672918    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:37.790241 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:37.790203    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0881fb93-59a0-4ea8-a982-5cc8e2f83b80" path="/var/lib/kubelet/pods/0881fb93-59a0-4ea8-a982-5cc8e2f83b80/volumes"
Oct 30 22:52:38.857592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:38.857550    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"]
Oct 30 22:52:39.237895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:39.237850    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:39.440554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:39.440519    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:41.227288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:41.227253    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:42.654608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:42.654570    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:42.918602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:42.918493    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:43.078100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:43.078058    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:43.110013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:43.109973    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"]
Oct 30 22:52:43.159913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:43.159878    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-37e79b-jqp28"]
Oct 30 22:52:43.558267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:43.558226    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:43.789123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:43.789090    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f35053-7342-4f66-bfe2-0190dea22b23" path="/var/lib/kubelet/pods/02f35053-7342-4f66-bfe2-0190dea22b23/volumes"
Oct 30 22:52:43.858497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:43.858400    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t" podUID="a26b5930-5b0b-4e6a-8d0f-344044a3c904" containerName="payload-processing" containerID="cri-o://ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c" gracePeriod=25
Oct 30 22:52:43.982112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:43.982066    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:44.116158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.116099    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:52:44.148078 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.148050    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:44.162140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.162112    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26b5930-5b0b-4e6a-8d0f-344044a3c904-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5930-5b0b-4e6a-8d0f-344044a3c904-plugins-config-volume\") pod \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") "
Oct 30 22:52:44.162235 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.162218    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26b5930-5b0b-4e6a-8d0f-344044a3c904-kube-api-access-2svd9\" (UniqueName: \"kubernetes.io/projected/a26b5930-5b0b-4e6a-8d0f-344044a3c904-kube-api-access-2svd9\") pod \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\" (UID: \"a26b5930-5b0b-4e6a-8d0f-344044a3c904\") "
Oct 30 22:52:44.162463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.162442    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26b5930-5b0b-4e6a-8d0f-344044a3c904-plugins-config-volume" pod "a26b5930-5b0b-4e6a-8d0f-344044a3c904" (UID: "a26b5930-5b0b-4e6a-8d0f-344044a3c904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:44.162619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.162601    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26b5930-5b0b-4e6a-8d0f-344044a3c904-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:44.164339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.164317    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26b5930-5b0b-4e6a-8d0f-344044a3c904-kube-api-access-2svd9" pod "a26b5930-5b0b-4e6a-8d0f-344044a3c904" (UID: "a26b5930-5b0b-4e6a-8d0f-344044a3c904"). InnerVolumeSpecName "kube-api-access-2svd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:44.263395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.263355    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2svd9\" (UniqueName: \"kubernetes.io/projected/a26b5930-5b0b-4e6a-8d0f-344044a3c904-kube-api-access-2svd9\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:44.498677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.498121    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:44.545399 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.545365    5503 generic.go:356] "Generic (PLEG): container finished" podID="a26b5930-5b0b-4e6a-8d0f-344044a3c904" containerID="ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c" exitCode=0
Oct 30 22:52:44.545575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.545404    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t" event={"ID":"a26b5930-5b0b-4e6a-8d0f-344044a3c904","Type":"ContainerDied","Data":"ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c"}
Oct 30 22:52:44.545575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.545453    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"
Oct 30 22:52:44.545575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.545545    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t" event={"ID":"a26b5930-5b0b-4e6a-8d0f-344044a3c904","Type":"ContainerDied","Data":"0bd0fb6a2e8232c2adff639f0c55f5d64a537b416e5a061c408bf182309c62d5"}
Oct 30 22:52:44.545575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.545575    5503 scope.go:122] "RemoveContainer" containerID="ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c"
Oct 30 22:52:44.555801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.555773    5503 scope.go:122] "RemoveContainer" containerID="ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c"
Oct 30 22:52:44.556051 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:44.556031    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c\": container with ID starting with ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c not found: ID does not exist" containerID="ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c"
Oct 30 22:52:44.556101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.556064    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c"} err="failed to get container status \"ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c\": rpc error: code = NotFound desc = could not find container \"ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c\": container with ID starting with ce7e7ec5ecd9cd8cafdd57ada0abe83ab9a4c6f35e3895b652be5428bf93734c not found: ID does not exist"
Oct 30 22:52:44.714249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.714212    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"]
Oct 30 22:52:44.764793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:44.764726    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-421286-677d995c69-fpl4t"]
Oct 30 22:52:45.254127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.254099    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"]
Oct 30 22:52:45.254586 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.254572    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a26b5930-5b0b-4e6a-8d0f-344044a3c904" containerName="payload-processing"
Oct 30 22:52:45.254657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.254587    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26b5930-5b0b-4e6a-8d0f-344044a3c904" containerName="payload-processing"
Oct 30 22:52:45.254708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.254699    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26b5930-5b0b-4e6a-8d0f-344044a3c904" containerName="payload-processing"
Oct 30 22:52:45.259141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.259125    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.311495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.311467    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"]
Oct 30 22:52:45.312550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.312520    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:45.312550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.312525    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:45.312741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.312662    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:45.359746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.359726    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:52:45.374563 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.374125    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.374893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.374866    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncz84\" (UniqueName: \"kubernetes.io/projected/30408336-cc07-4806-8ecf-fdbfc19deaaf-kube-api-access-ncz84\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.375041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.375021    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.476179 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.476141    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.476345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.476198    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.476345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.476264    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncz84\" (UniqueName: \"kubernetes.io/projected/30408336-cc07-4806-8ecf-fdbfc19deaaf-kube-api-access-ncz84\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.476845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.476820    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.478587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.478565    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.540942 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.540879    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncz84\" (UniqueName: \"kubernetes.io/projected/30408336-cc07-4806-8ecf-fdbfc19deaaf-kube-api-access-ncz84\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.591136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.591111    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:45.789039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.789005    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26b5930-5b0b-4e6a-8d0f-344044a3c904" path="/var/lib/kubelet/pods/a26b5930-5b0b-4e6a-8d0f-344044a3c904/volumes"
Oct 30 22:52:45.962082 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:52:45.962052    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30408336_cc07_4806_8ecf_fdbfc19deaaf.slice/crio-c2c92cdff5bed6067655d19a031c755e9d662c2538fcc5a4ef49b485e517633e WatchSource:0}: Error finding container c2c92cdff5bed6067655d19a031c755e9d662c2538fcc5a4ef49b485e517633e: Status 404 returned error can't find the container with id c2c92cdff5bed6067655d19a031c755e9d662c2538fcc5a4ef49b485e517633e
Oct 30 22:52:45.964866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:45.964840    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"]
Oct 30 22:52:46.560469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:46.560393    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h" event={"ID":"30408336-cc07-4806-8ecf-fdbfc19deaaf","Type":"ContainerStarted","Data":"a6e4aae3bc14cc4c432d566ce849430a210c7c86c51e2ebf4ae9c6c8001ac046"}
Oct 30 22:52:46.560469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:46.560453    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h" event={"ID":"30408336-cc07-4806-8ecf-fdbfc19deaaf","Type":"ContainerStarted","Data":"c2c92cdff5bed6067655d19a031c755e9d662c2538fcc5a4ef49b485e517633e"}
Oct 30 22:52:46.692281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:46.692234    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h" podStartSLOduration=1.692221362 podStartE2EDuration="1.692221362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:46.69126859 +0000 UTC m=+3943.025845358" watchObservedRunningTime="2026-10-30 22:52:46.692221362 +0000 UTC m=+3943.026798168"
Oct 30 22:52:47.278726 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:47.278690    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:52:47.502075 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:47.502044    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:52:47.567720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:47.567621    5503 generic.go:356] "Generic (PLEG): container finished" podID="30408336-cc07-4806-8ecf-fdbfc19deaaf" containerID="a6e4aae3bc14cc4c432d566ce849430a210c7c86c51e2ebf4ae9c6c8001ac046" exitCode=0
Oct 30 22:52:47.567720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:47.567707    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h" event={"ID":"30408336-cc07-4806-8ecf-fdbfc19deaaf","Type":"ContainerDied","Data":"a6e4aae3bc14cc4c432d566ce849430a210c7c86c51e2ebf4ae9c6c8001ac046"}
Oct 30 22:52:47.777932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:47.777900    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:48.719345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.719318    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:48.808114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.808086    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-xks-ca\") pod \"30408336-cc07-4806-8ecf-fdbfc19deaaf\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") "
Oct 30 22:52:48.808276 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.808161    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-openshift-ca\") pod \"30408336-cc07-4806-8ecf-fdbfc19deaaf\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") "
Oct 30 22:52:48.808327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.808280    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30408336-cc07-4806-8ecf-fdbfc19deaaf-kube-api-access-ncz84\" (UniqueName: \"kubernetes.io/projected/30408336-cc07-4806-8ecf-fdbfc19deaaf-kube-api-access-ncz84\") pod \"30408336-cc07-4806-8ecf-fdbfc19deaaf\" (UID: \"30408336-cc07-4806-8ecf-fdbfc19deaaf\") "
Oct 30 22:52:48.808548 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.808510    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-openshift-ca" pod "30408336-cc07-4806-8ecf-fdbfc19deaaf" (UID: "30408336-cc07-4806-8ecf-fdbfc19deaaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:48.810340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.810297    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30408336-cc07-4806-8ecf-fdbfc19deaaf-kube-api-access-ncz84" pod "30408336-cc07-4806-8ecf-fdbfc19deaaf" (UID: "30408336-cc07-4806-8ecf-fdbfc19deaaf"). InnerVolumeSpecName "kube-api-access-ncz84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:48.810432 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.810365    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-xks-ca" pod "30408336-cc07-4806-8ecf-fdbfc19deaaf" (UID: "30408336-cc07-4806-8ecf-fdbfc19deaaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:48.909578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.909548    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:48.909772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.909607    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30408336-cc07-4806-8ecf-fdbfc19deaaf-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:48.909772 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:48.909618    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncz84\" (UniqueName: \"kubernetes.io/projected/30408336-cc07-4806-8ecf-fdbfc19deaaf-kube-api-access-ncz84\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.586442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:49.586411    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"
Oct 30 22:52:49.586442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:49.586430    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h" event={"ID":"30408336-cc07-4806-8ecf-fdbfc19deaaf","Type":"ContainerDied","Data":"c2c92cdff5bed6067655d19a031c755e9d662c2538fcc5a4ef49b485e517633e"}
Oct 30 22:52:49.586698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:49.586496    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c92cdff5bed6067655d19a031c755e9d662c2538fcc5a4ef49b485e517633e"
Oct 30 22:52:49.917826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:49.917789    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.078094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:50.078064    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.260418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:50.260340    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:50.755685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:50.755623    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:51.287019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.286988    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"]
Oct 30 22:52:51.287731 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.287712    5503 cpu_manager.go:414] "containerMap: removing container" podUID="30408336-cc07-4806-8ecf-fdbfc19deaaf" containerName="revoke-keys"
Oct 30 22:52:51.287807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.287734    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30408336-cc07-4806-8ecf-fdbfc19deaaf" containerName="revoke-keys"
Oct 30 22:52:51.287918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.287907    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="30408336-cc07-4806-8ecf-fdbfc19deaaf" containerName="revoke-keys"
Oct 30 22:52:51.291619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.291599    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.335686 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.335648    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntgv\" (UniqueName: \"kubernetes.io/projected/0204fecc-b36b-46a3-b299-5573da2237bd-kube-api-access-bntgv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.335846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.335730    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.335932 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.335845    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.343800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.343771    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"]
Oct 30 22:52:51.392016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.391989    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:51.392016 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.392005    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:52:51.392218 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.392047    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:52:51.396769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.396753    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:52:51.437135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.437104    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntgv\" (UniqueName: \"kubernetes.io/projected/0204fecc-b36b-46a3-b299-5573da2237bd-kube-api-access-bntgv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.437322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.437147    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.437322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.437211    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.437867 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.437846    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.439532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.439510    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.499690 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.499661    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntgv\" (UniqueName: \"kubernetes.io/projected/0204fecc-b36b-46a3-b299-5573da2237bd-kube-api-access-bntgv\") pod \"maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.605983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.605898    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:51.978654 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:52:51.978605    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0204fecc_b36b_46a3_b299_5573da2237bd.slice/crio-b7ab85e01cf6c28f5f664586514578398998bfc6a406fed2ab05ab9b39c349f4 WatchSource:0}: Error finding container b7ab85e01cf6c28f5f664586514578398998bfc6a406fed2ab05ab9b39c349f4: Status 404 returned error can't find the container with id b7ab85e01cf6c28f5f664586514578398998bfc6a406fed2ab05ab9b39c349f4
Oct 30 22:52:51.980809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:51.980777    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"]
Oct 30 22:52:52.608273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:52.608182    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k" event={"ID":"0204fecc-b36b-46a3-b299-5573da2237bd","Type":"ContainerStarted","Data":"e175993ee61deb6363b39be7225d35f53b513fc671e1e015beda791666b27b94"}
Oct 30 22:52:52.608273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:52.608249    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k" event={"ID":"0204fecc-b36b-46a3-b299-5573da2237bd","Type":"ContainerStarted","Data":"b7ab85e01cf6c28f5f664586514578398998bfc6a406fed2ab05ab9b39c349f4"}
Oct 30 22:52:52.645532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:52.645498    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:52.727889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:52.727838    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k" podStartSLOduration=1.72782419 podStartE2EDuration="1.72782419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:52.725756467 +0000 UTC m=+3949.060333247" watchObservedRunningTime="2026-10-30 22:52:52.72782419 +0000 UTC m=+3949.062400960"
Oct 30 22:52:53.617624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:53.617588    5503 generic.go:356] "Generic (PLEG): container finished" podID="0204fecc-b36b-46a3-b299-5573da2237bd" containerID="e175993ee61deb6363b39be7225d35f53b513fc671e1e015beda791666b27b94" exitCode=0
Oct 30 22:52:53.618054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:53.617675    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k" event={"ID":"0204fecc-b36b-46a3-b299-5573da2237bd","Type":"ContainerDied","Data":"e175993ee61deb6363b39be7225d35f53b513fc671e1e015beda791666b27b94"}
Oct 30 22:52:54.762076 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.762051    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:54.877454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.877364    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0204fecc-b36b-46a3-b299-5573da2237bd-kube-api-access-bntgv\" (UniqueName: \"kubernetes.io/projected/0204fecc-b36b-46a3-b299-5573da2237bd-kube-api-access-bntgv\") pod \"0204fecc-b36b-46a3-b299-5573da2237bd\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") "
Oct 30 22:52:54.877671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.877463    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-xks-ca\") pod \"0204fecc-b36b-46a3-b299-5573da2237bd\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") "
Oct 30 22:52:54.877671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.877565    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-openshift-ca\") pod \"0204fecc-b36b-46a3-b299-5573da2237bd\" (UID: \"0204fecc-b36b-46a3-b299-5573da2237bd\") "
Oct 30 22:52:54.878006 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.877979    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-openshift-ca" pod "0204fecc-b36b-46a3-b299-5573da2237bd" (UID: "0204fecc-b36b-46a3-b299-5573da2237bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:54.878146 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.878118    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:54.879577 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.879555    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-xks-ca" pod "0204fecc-b36b-46a3-b299-5573da2237bd" (UID: "0204fecc-b36b-46a3-b299-5573da2237bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:54.879721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.879700    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0204fecc-b36b-46a3-b299-5573da2237bd-kube-api-access-bntgv" pod "0204fecc-b36b-46a3-b299-5573da2237bd" (UID: "0204fecc-b36b-46a3-b299-5573da2237bd"). InnerVolumeSpecName "kube-api-access-bntgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:54.979005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.978971    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0204fecc-b36b-46a3-b299-5573da2237bd-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:54.979005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:54.979009    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bntgv\" (UniqueName: \"kubernetes.io/projected/0204fecc-b36b-46a3-b299-5573da2237bd-kube-api-access-bntgv\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:55.631838 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:55.631801    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"
Oct 30 22:52:55.632026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:55.631838    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k" event={"ID":"0204fecc-b36b-46a3-b299-5573da2237bd","Type":"ContainerDied","Data":"b7ab85e01cf6c28f5f664586514578398998bfc6a406fed2ab05ab9b39c349f4"}
Oct 30 22:52:55.632026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:55.631889    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ab85e01cf6c28f5f664586514578398998bfc6a406fed2ab05ab9b39c349f4"
Oct 30 22:52:55.746938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:55.746905    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"]
Oct 30 22:52:55.846766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:55.846733    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:56.940103 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:56.940069    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"]
Oct 30 22:52:56.957928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:56.957893    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:57.213213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:57.213130    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:57.362232 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:57.362198    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:52:58.098028 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:52:58.097987    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:52:58.783331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:52:58.783301    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:01.381568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:01.381531    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:01.659677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:01.659645    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:01.848500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:01.848467    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"]
Oct 30 22:53:01.899274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:01.899239    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fc6466-4s28h"]
Oct 30 22:53:01.941042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:01.940940    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" podUID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" containerName="payload-pre-processing" containerID="cri-o://654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06" gracePeriod=25
Oct 30 22:53:02.314670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:02.314559    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:03.432352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:03.432320    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"]
Oct 30 22:53:03.588827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:03.588785    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:03.789040 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:03.788965    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30408336-cc07-4806-8ecf-fdbfc19deaaf" path="/var/lib/kubelet/pods/30408336-cc07-4806-8ecf-fdbfc19deaaf/volumes"
Oct 30 22:53:03.821753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:03.821723    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:03.987015 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:03.986978    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:08.322058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:08.322018    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:08.433293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:08.433249    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" podUID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" containerName="payload-processing" containerID="cri-o://f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf" gracePeriod=25
Oct 30 22:53:08.481330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:08.481280    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:08.782560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:08.782531    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:09.666028 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.665987    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" podUID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:09.668428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.668403    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"]
Oct 30 22:53:09.668893 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.668880    5503 cpu_manager.go:414] "containerMap: removing container" podUID="0204fecc-b36b-46a3-b299-5573da2237bd" containerName="revoke-keys"
Oct 30 22:53:09.668936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.668894    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0204fecc-b36b-46a3-b299-5573da2237bd" containerName="revoke-keys"
Oct 30 22:53:09.669017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.669008    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="0204fecc-b36b-46a3-b299-5573da2237bd" containerName="revoke-keys"
Oct 30 22:53:09.672274 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.672258    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.774702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.774669    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:09.777046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.777025    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"]
Oct 30 22:53:09.777263 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.777242    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:09.777403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.777269    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:09.777403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.777379    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:53:09.846393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.846355    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.846393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.846396    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzqx4\" (UniqueName: \"kubernetes.io/projected/aad2746d-9ef0-46de-ac7b-7a6c450a7965-kube-api-access-fzqx4\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.846676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.846516    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.947558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.947465    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.947764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.947619    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.947764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.947697    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzqx4\" (UniqueName: \"kubernetes.io/projected/aad2746d-9ef0-46de-ac7b-7a6c450a7965-kube-api-access-fzqx4\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.948262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.948242    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:09.949986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:09.949967    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:10.010924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:10.010899    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzqx4\" (UniqueName: \"kubernetes.io/projected/aad2746d-9ef0-46de-ac7b-7a6c450a7965-kube-api-access-fzqx4\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:10.284672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:10.284580    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:10.658341 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:53:10.658303    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad2746d_9ef0_46de_ac7b_7a6c450a7965.slice/crio-054e545f1ffe862e168722b82e65092e1beefcc0f17b87ebfb4660263f6e641a WatchSource:0}: Error finding container 054e545f1ffe862e168722b82e65092e1beefcc0f17b87ebfb4660263f6e641a: Status 404 returned error can't find the container with id 054e545f1ffe862e168722b82e65092e1beefcc0f17b87ebfb4660263f6e641a
Oct 30 22:53:10.660708 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:10.660679    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"]
Oct 30 22:53:10.745344 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:10.745316    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn" event={"ID":"aad2746d-9ef0-46de-ac7b-7a6c450a7965","Type":"ContainerStarted","Data":"054e545f1ffe862e168722b82e65092e1beefcc0f17b87ebfb4660263f6e641a"}
Oct 30 22:53:10.767239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:10.767210    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main" containerID="cri-o://07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2" gracePeriod=45
Oct 30 22:53:10.858716 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:10.858678    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 30 22:53:11.014002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.013974    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:53:11.162328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162239    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-home\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-home\") pod \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") "
Oct 30 22:53:11.162507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162400    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-kube-api-access-7gtjq\" (UniqueName: \"kubernetes.io/projected/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-kube-api-access-7gtjq\") pod \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") "
Oct 30 22:53:11.162507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162441    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tmp-dir\") pod \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") "
Oct 30 22:53:11.162507 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162481    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-home" pod "dccedd2d-6a0b-4c71-bb7c-eda4063e6092" (UID: "dccedd2d-6a0b-4c71-bb7c-eda4063e6092"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:11.162706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162488    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-model-cache\") pod \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") "
Oct 30 22:53:11.162706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162625    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-dshm\") pod \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") "
Oct 30 22:53:11.162840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162773    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-model-cache" pod "dccedd2d-6a0b-4c71-bb7c-eda4063e6092" (UID: "dccedd2d-6a0b-4c71-bb7c-eda4063e6092"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:11.162840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.162802    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tmp-dir" pod "dccedd2d-6a0b-4c71-bb7c-eda4063e6092" (UID: "dccedd2d-6a0b-4c71-bb7c-eda4063e6092"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:11.163141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.163113    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tls-certs\" (UniqueName: \"kubernetes.io/secret/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tls-certs\") pod \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\" (UID: \"dccedd2d-6a0b-4c71-bb7c-eda4063e6092\") "
Oct 30 22:53:11.164087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.164047    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:11.164087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.164071    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:11.164337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.164111    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:11.165256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.165219    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-kube-api-access-7gtjq" pod "dccedd2d-6a0b-4c71-bb7c-eda4063e6092" (UID: "dccedd2d-6a0b-4c71-bb7c-eda4063e6092"). InnerVolumeSpecName "kube-api-access-7gtjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:11.165777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.165750    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-dshm" pod "dccedd2d-6a0b-4c71-bb7c-eda4063e6092" (UID: "dccedd2d-6a0b-4c71-bb7c-eda4063e6092"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:11.165974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.165957    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tls-certs" pod "dccedd2d-6a0b-4c71-bb7c-eda4063e6092" (UID: "dccedd2d-6a0b-4c71-bb7c-eda4063e6092"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:11.265522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.265492    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:11.265522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.265519    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:11.265522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.265528    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gtjq\" (UniqueName: \"kubernetes.io/projected/dccedd2d-6a0b-4c71-bb7c-eda4063e6092-kube-api-access-7gtjq\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:11.753042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.752964    5503 generic.go:356] "Generic (PLEG): container finished" podID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerID="07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2" exitCode=0
Oct 30 22:53:11.753042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.753009    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"
Oct 30 22:53:11.753553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.753041    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" event={"ID":"dccedd2d-6a0b-4c71-bb7c-eda4063e6092","Type":"ContainerDied","Data":"07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2"}
Oct 30 22:53:11.753553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.753082    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw" event={"ID":"dccedd2d-6a0b-4c71-bb7c-eda4063e6092","Type":"ContainerDied","Data":"23ac3756e3c643723f6e33644af4cb8afe9c38ce379c559dbd7ccb46755db9c0"}
Oct 30 22:53:11.753553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.753098    5503 scope.go:122] "RemoveContainer" containerID="07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2"
Oct 30 22:53:11.754807 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.754774    5503 generic.go:356] "Generic (PLEG): container finished" podID="aad2746d-9ef0-46de-ac7b-7a6c450a7965" containerID="a1d12f3482ea36abc2fbb119e40969ff689e97a8efd458e5ff1584d5ba4611da" exitCode=0
Oct 30 22:53:11.754933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.754829    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn" event={"ID":"aad2746d-9ef0-46de-ac7b-7a6c450a7965","Type":"ContainerDied","Data":"a1d12f3482ea36abc2fbb119e40969ff689e97a8efd458e5ff1584d5ba4611da"}
Oct 30 22:53:11.763046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.763014    5503 scope.go:122] "RemoveContainer" containerID="07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2"
Oct 30 22:53:11.763338 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:53:11.763318    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2\": container with ID starting with 07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2 not found: ID does not exist" containerID="07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2"
Oct 30 22:53:11.763416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.763348    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2"} err="failed to get container status \"07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2\": rpc error: code = NotFound desc = could not find container \"07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2\": container with ID starting with 07048aaa5d7ffe7d0e9a48de585ec336bcfdb2581891cb4512293dd9d4e4bba2 not found: ID does not exist"
Oct 30 22:53:11.926289 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.926238    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"]
Oct 30 22:53:11.982832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:11.979783    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e524ce/premium-w4-e524ce-kserve-569fb76f98-nvzsw"]
Oct 30 22:53:12.895512 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:12.895484    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:12.984622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:12.984595    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad2746d-9ef0-46de-ac7b-7a6c450a7965-kube-api-access-fzqx4\" (UniqueName: \"kubernetes.io/projected/aad2746d-9ef0-46de-ac7b-7a6c450a7965-kube-api-access-fzqx4\") pod \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") "
Oct 30 22:53:12.984802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:12.984731    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-xks-ca\") pod \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") "
Oct 30 22:53:12.984802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:12.984756    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-openshift-ca\") pod \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\" (UID: \"aad2746d-9ef0-46de-ac7b-7a6c450a7965\") "
Oct 30 22:53:12.985163 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:12.985137    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-openshift-ca" pod "aad2746d-9ef0-46de-ac7b-7a6c450a7965" (UID: "aad2746d-9ef0-46de-ac7b-7a6c450a7965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:12.986665 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:12.986627    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-xks-ca" pod "aad2746d-9ef0-46de-ac7b-7a6c450a7965" (UID: "aad2746d-9ef0-46de-ac7b-7a6c450a7965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:12.986768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:12.986680    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad2746d-9ef0-46de-ac7b-7a6c450a7965-kube-api-access-fzqx4" pod "aad2746d-9ef0-46de-ac7b-7a6c450a7965" (UID: "aad2746d-9ef0-46de-ac7b-7a6c450a7965"). InnerVolumeSpecName "kube-api-access-fzqx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:13.086469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.086396    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:13.086469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.086423    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad2746d-9ef0-46de-ac7b-7a6c450a7965-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:13.086469 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.086434    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzqx4\" (UniqueName: \"kubernetes.io/projected/aad2746d-9ef0-46de-ac7b-7a6c450a7965-kube-api-access-fzqx4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:13.123288 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.123254    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:13.769176 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.769145    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"
Oct 30 22:53:13.769381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.769175    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn" event={"ID":"aad2746d-9ef0-46de-ac7b-7a6c450a7965","Type":"ContainerDied","Data":"054e545f1ffe862e168722b82e65092e1beefcc0f17b87ebfb4660263f6e641a"}
Oct 30 22:53:13.769381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.769261    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054e545f1ffe862e168722b82e65092e1beefcc0f17b87ebfb4660263f6e641a"
Oct 30 22:53:13.789495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:13.789466    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" path="/var/lib/kubelet/pods/dccedd2d-6a0b-4c71-bb7c-eda4063e6092/volumes"
Oct 30 22:53:14.099849 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:14.099773    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" podUID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:17.901359 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:17.901322    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:19.666290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:19.666255    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" podUID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:21.640660 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:21.640586    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:21.827129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:21.827091    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:22.204959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.204888    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:53:22.277245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.277213    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54e2f66-cb0e-41e5-af46-3fb7948dd396-kube-api-access-7vlz7\" (UniqueName: \"kubernetes.io/projected/a54e2f66-cb0e-41e5-af46-3fb7948dd396-kube-api-access-7vlz7\") pod \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") "
Oct 30 22:53:22.277448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.277267    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54e2f66-cb0e-41e5-af46-3fb7948dd396-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54e2f66-cb0e-41e5-af46-3fb7948dd396-plugins-config-volume\") pod \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\" (UID: \"a54e2f66-cb0e-41e5-af46-3fb7948dd396\") "
Oct 30 22:53:22.277761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.277735    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54e2f66-cb0e-41e5-af46-3fb7948dd396-plugins-config-volume" pod "a54e2f66-cb0e-41e5-af46-3fb7948dd396" (UID: "a54e2f66-cb0e-41e5-af46-3fb7948dd396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:22.279395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.279364    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54e2f66-cb0e-41e5-af46-3fb7948dd396-kube-api-access-7vlz7" pod "a54e2f66-cb0e-41e5-af46-3fb7948dd396" (UID: "a54e2f66-cb0e-41e5-af46-3fb7948dd396"). InnerVolumeSpecName "kube-api-access-7vlz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:22.379356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.379313    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlz7\" (UniqueName: \"kubernetes.io/projected/a54e2f66-cb0e-41e5-af46-3fb7948dd396-kube-api-access-7vlz7\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:22.379356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.379350    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54e2f66-cb0e-41e5-af46-3fb7948dd396-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:22.837342 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.837302    5503 generic.go:356] "Generic (PLEG): container finished" podID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" containerID="654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06" exitCode=0
Oct 30 22:53:22.837835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.837369    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"
Oct 30 22:53:22.837835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.837383    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" event={"ID":"a54e2f66-cb0e-41e5-af46-3fb7948dd396","Type":"ContainerDied","Data":"654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06"}
Oct 30 22:53:22.837835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.837471    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg" event={"ID":"a54e2f66-cb0e-41e5-af46-3fb7948dd396","Type":"ContainerDied","Data":"fc87a4fcfb54d5b09524b632c1efe4b9d025b1d796f7b33ccfb93065a08a5135"}
Oct 30 22:53:22.837835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.837496    5503 scope.go:122] "RemoveContainer" containerID="654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06"
Oct 30 22:53:22.847422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.847399    5503 scope.go:122] "RemoveContainer" containerID="654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06"
Oct 30 22:53:22.847748 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:53:22.847720    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06\": container with ID starting with 654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06 not found: ID does not exist" containerID="654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06"
Oct 30 22:53:22.847855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.847757    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06"} err="failed to get container status \"654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06\": rpc error: code = NotFound desc = could not find container \"654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06\": container with ID starting with 654ce6afbda9cedb7bdd049c80e02b2196479dcd8f324a993c90a81df86a2c06 not found: ID does not exist"
Oct 30 22:53:22.878925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.878865    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:22.997389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:22.997349    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"]
Oct 30 22:53:23.042957 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:23.042927    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc6466-6995fc8bc5-fdghg"]
Oct 30 22:53:23.790150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:23.790116    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" path="/var/lib/kubelet/pods/a54e2f66-cb0e-41e5-af46-3fb7948dd396/volumes"
Oct 30 22:53:24.099101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:24.099022    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" podUID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:27.622933 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:27.622897    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:28.098658 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:53:28.098602    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:53:28.499608 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:53:28.499567    5503 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9bcd58_5e7f_482b_b61e_06fcd5882a4b.slice/crio-conmon-f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:53:28.692364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.692300    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:53:28.754149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.754118    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-kube-api-access-vwfj5\" (UniqueName: \"kubernetes.io/projected/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-kube-api-access-vwfj5\") pod \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") "
Oct 30 22:53:28.754343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.754326    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-plugins-config-volume\") pod \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\" (UID: \"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b\") "
Oct 30 22:53:28.754688 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.754668    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-plugins-config-volume" pod "bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" (UID: "bc9bcd58-5e7f-482b-b61e-06fcd5882a4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:28.756237 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.756215    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-kube-api-access-vwfj5" pod "bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" (UID: "bc9bcd58-5e7f-482b-b61e-06fcd5882a4b"). InnerVolumeSpecName "kube-api-access-vwfj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:28.856117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.856074    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.856117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.856107    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwfj5\" (UniqueName: \"kubernetes.io/projected/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b-kube-api-access-vwfj5\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.877652 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.877607    5503 generic.go:356] "Generic (PLEG): container finished" podID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" containerID="f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf" exitCode=0
Oct 30 22:53:28.877837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.877668    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"
Oct 30 22:53:28.877837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.877689    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" event={"ID":"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b","Type":"ContainerDied","Data":"f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf"}
Oct 30 22:53:28.877837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.877749    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd" event={"ID":"bc9bcd58-5e7f-482b-b61e-06fcd5882a4b","Type":"ContainerDied","Data":"52b0468e9a939deea518166a17a43c65c8013100ef3d46a7fb4112a5d1434b9f"}
Oct 30 22:53:28.877837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.877774    5503 scope.go:122] "RemoveContainer" containerID="f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf"
Oct 30 22:53:28.888097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.888081    5503 scope.go:122] "RemoveContainer" containerID="f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf"
Oct 30 22:53:28.888392 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:53:28.888364    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf\": container with ID starting with f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf not found: ID does not exist" containerID="f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf"
Oct 30 22:53:28.888504 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:28.888398    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf"} err="failed to get container status \"f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf\": rpc error: code = NotFound desc = could not find container \"f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf\": container with ID starting with f4446b8f33f97e96976b97cb768ecaae9c97379dc473ae3367474cec625dbbaf not found: ID does not exist"
Oct 30 22:53:29.035255 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:29.035170    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"]
Oct 30 22:53:29.039564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:29.039536    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e524ce-7f6847f7bc-flnfd"]
Oct 30 22:53:29.788522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:29.788484    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" path="/var/lib/kubelet/pods/bc9bcd58-5e7f-482b-b61e-06fcd5882a4b/volumes"
Oct 30 22:53:31.620483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:31.620452    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:32.002682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:32.002647    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:32.154048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:32.153981    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:32.339801 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:32.339703    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:33.169538 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:33.169500    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:33.437378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:33.437281    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:33.586736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:33.586441    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:33.669559 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:33.669523    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"]
Oct 30 22:53:33.715539 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:33.715459    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-421286-pj6pf"]
Oct 30 22:53:33.737010 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:33.736978    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:33.789135 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:33.789093    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c668030-43ca-4cb6-8bbb-b4313761ed81" path="/var/lib/kubelet/pods/3c668030-43ca-4cb6-8bbb-b4313761ed81/volumes"
Oct 30 22:53:34.083071 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:34.082996    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:34.448777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:34.448745    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:35.101177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:35.101135    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:53:38.359260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:38.359219    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:40.061700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:40.061663    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:40.210178 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:40.210149    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:40.403859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:40.403819    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:40.902524 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:40.902491    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:41.056284 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.056249    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:41.237713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.237621    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"]
Oct 30 22:53:41.238101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238070    5503 cpu_manager.go:414] "containerMap: removing container" podUID="aad2746d-9ef0-46de-ac7b-7a6c450a7965" containerName="revoke-keys"
Oct 30 22:53:41.238101 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238081    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad2746d-9ef0-46de-ac7b-7a6c450a7965" containerName="revoke-keys"
Oct 30 22:53:41.238173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238106    5503 cpu_manager.go:414] "containerMap: removing container" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main"
Oct 30 22:53:41.238173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238112    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main"
Oct 30 22:53:41.238173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238118    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" containerName="payload-pre-processing"
Oct 30 22:53:41.238173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238123    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" containerName="payload-pre-processing"
Oct 30 22:53:41.238173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238139    5503 cpu_manager.go:414] "containerMap: removing container" podUID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" containerName="payload-processing"
Oct 30 22:53:41.238173 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238143    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" containerName="payload-processing"
Oct 30 22:53:41.238416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238217    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccedd2d-6a0b-4c71-bb7c-eda4063e6092" containerName="main"
Oct 30 22:53:41.238416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238224    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54e2f66-cb0e-41e5-af46-3fb7948dd396" containerName="payload-pre-processing"
Oct 30 22:53:41.238416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238234    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9bcd58-5e7f-482b-b61e-06fcd5882a4b" containerName="payload-processing"
Oct 30 22:53:41.238416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.238244    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad2746d-9ef0-46de-ac7b-7a6c450a7965" containerName="revoke-keys"
Oct 30 22:53:41.241534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.241513    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.293515 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.293462    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:41.340111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.340070    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:41.340111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.340105    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:41.340339 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.340296    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:53:41.342759 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.342741    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"]
Oct 30 22:53:41.397605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.397568    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.397802 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.397718    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48br\" (UniqueName: \"kubernetes.io/projected/14621d87-6e5c-4074-a8ed-b4728e749832-kube-api-access-d48br\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.397873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.397805    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.498769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.498683    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48br\" (UniqueName: \"kubernetes.io/projected/14621d87-6e5c-4074-a8ed-b4728e749832-kube-api-access-d48br\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.498769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.498735    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.498983 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.498809    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.499482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.499454    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.501035 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.501017    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.557450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.557416    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48br\" (UniqueName: \"kubernetes.io/projected/14621d87-6e5c-4074-a8ed-b4728e749832-kube-api-access-d48br\") pod \"maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:41.855712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:41.855607    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:42.231553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:42.231522    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"]
Oct 30 22:53:42.233134 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:53:42.233102    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14621d87_6e5c_4074_a8ed_b4728e749832.slice/crio-83fbb4027dacb73e42ad27bb04f6b88cd88dcfee17f50f7fd16ffac95f3d3e9d WatchSource:0}: Error finding container 83fbb4027dacb73e42ad27bb04f6b88cd88dcfee17f50f7fd16ffac95f3d3e9d: Status 404 returned error can't find the container with id 83fbb4027dacb73e42ad27bb04f6b88cd88dcfee17f50f7fd16ffac95f3d3e9d
Oct 30 22:53:42.987706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:42.987667    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj" event={"ID":"14621d87-6e5c-4074-a8ed-b4728e749832","Type":"ContainerStarted","Data":"c5238e54a718c70e78ed29557ac33ee5b3dd4cc0ef5b4537e0cd15e9718735de"}
Oct 30 22:53:42.988096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:42.987728    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj" event={"ID":"14621d87-6e5c-4074-a8ed-b4728e749832","Type":"ContainerStarted","Data":"83fbb4027dacb73e42ad27bb04f6b88cd88dcfee17f50f7fd16ffac95f3d3e9d"}
Oct 30 22:53:43.099198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:43.099150    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj" podStartSLOduration=2.099137168 podStartE2EDuration="2.099137168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:43.097217282 +0000 UTC m=+3999.431794064" watchObservedRunningTime="2026-10-30 22:53:43.099137168 +0000 UTC m=+3999.433713937"
Oct 30 22:53:43.142720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:43.142681    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:43.995692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:43.995659    5503 generic.go:356] "Generic (PLEG): container finished" podID="14621d87-6e5c-4074-a8ed-b4728e749832" containerID="c5238e54a718c70e78ed29557ac33ee5b3dd4cc0ef5b4537e0cd15e9718735de" exitCode=0
Oct 30 22:53:43.996445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:43.995726    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj" event={"ID":"14621d87-6e5c-4074-a8ed-b4728e749832","Type":"ContainerDied","Data":"c5238e54a718c70e78ed29557ac33ee5b3dd4cc0ef5b4537e0cd15e9718735de"}
Oct 30 22:53:45.140542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.140510    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:45.237597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.237558    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-xks-ca\") pod \"14621d87-6e5c-4074-a8ed-b4728e749832\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") "
Oct 30 22:53:45.237831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.237691    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14621d87-6e5c-4074-a8ed-b4728e749832-kube-api-access-d48br\" (UniqueName: \"kubernetes.io/projected/14621d87-6e5c-4074-a8ed-b4728e749832-kube-api-access-d48br\") pod \"14621d87-6e5c-4074-a8ed-b4728e749832\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") "
Oct 30 22:53:45.237831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.237737    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-openshift-ca\") pod \"14621d87-6e5c-4074-a8ed-b4728e749832\" (UID: \"14621d87-6e5c-4074-a8ed-b4728e749832\") "
Oct 30 22:53:45.238124 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.238097    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-openshift-ca" pod "14621d87-6e5c-4074-a8ed-b4728e749832" (UID: "14621d87-6e5c-4074-a8ed-b4728e749832"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:45.239652 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.239604    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-xks-ca" pod "14621d87-6e5c-4074-a8ed-b4728e749832" (UID: "14621d87-6e5c-4074-a8ed-b4728e749832"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:45.239789 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.239770    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14621d87-6e5c-4074-a8ed-b4728e749832-kube-api-access-d48br" pod "14621d87-6e5c-4074-a8ed-b4728e749832" (UID: "14621d87-6e5c-4074-a8ed-b4728e749832"). InnerVolumeSpecName "kube-api-access-d48br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:45.339375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.339298    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d48br\" (UniqueName: \"kubernetes.io/projected/14621d87-6e5c-4074-a8ed-b4728e749832-kube-api-access-d48br\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:45.339375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.339327    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:45.339375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:45.339337    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14621d87-6e5c-4074-a8ed-b4728e749832-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:46.009498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:46.009470    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"
Oct 30 22:53:46.009498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:46.009488    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj" event={"ID":"14621d87-6e5c-4074-a8ed-b4728e749832","Type":"ContainerDied","Data":"83fbb4027dacb73e42ad27bb04f6b88cd88dcfee17f50f7fd16ffac95f3d3e9d"}
Oct 30 22:53:46.009732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:46.009530    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83fbb4027dacb73e42ad27bb04f6b88cd88dcfee17f50f7fd16ffac95f3d3e9d"
Oct 30 22:53:47.245950 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.245912    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"]
Oct 30 22:53:47.246449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.246412    5503 cpu_manager.go:414] "containerMap: removing container" podUID="14621d87-6e5c-4074-a8ed-b4728e749832" containerName="revoke-keys"
Oct 30 22:53:47.246449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.246423    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14621d87-6e5c-4074-a8ed-b4728e749832" containerName="revoke-keys"
Oct 30 22:53:47.246565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.246509    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="14621d87-6e5c-4074-a8ed-b4728e749832" containerName="revoke-keys"
Oct 30 22:53:47.249903 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.249882    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.291943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.291909    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"]
Oct 30 22:53:47.295924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.295899    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.336529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.336497    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:53:47.336529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.336502    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:47.336529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.336504    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:53:47.339039 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.339018    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"]
Oct 30 22:53:47.349332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.349304    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:53:47.352116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.352089    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"]
Oct 30 22:53:47.362338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.362311    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.362549 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.362529    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwb4n\" (UniqueName: \"kubernetes.io/projected/a5e43d4f-b1f5-4502-bae8-827e24270207-kube-api-access-zwb4n\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.362682 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.362663    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.463963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.463928    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.464153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.463973    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwb4n\" (UniqueName: \"kubernetes.io/projected/a5e43d4f-b1f5-4502-bae8-827e24270207-kube-api-access-zwb4n\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.464153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.464070    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.464153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.464135    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.464318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.464170    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.464318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.464195    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5vx9\" (UniqueName: \"kubernetes.io/projected/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-kube-api-access-v5vx9\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.464874 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.464849    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.466378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.466359    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.523738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.523628    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwb4n\" (UniqueName: \"kubernetes.io/projected/a5e43d4f-b1f5-4502-bae8-827e24270207-kube-api-access-zwb4n\") pod \"maas-api-revoke-sub-9a3d0123-zsh72\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.562607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.562572    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:47.565201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.565181    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.565298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.565275    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.565337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.565312    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5vx9\" (UniqueName: \"kubernetes.io/projected/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-kube-api-access-v5vx9\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.566175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.566033    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.567667 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.567622    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.656099 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.656031    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5vx9\" (UniqueName: \"kubernetes.io/projected/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-kube-api-access-v5vx9\") pod \"maas-api-revoke-sub-59f77dfe-wb2hx\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.912914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.912879    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:47.946175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:47.946147    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"]
Oct 30 22:53:47.953791 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:53:47.953757    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e43d4f_b1f5_4502_bae8_827e24270207.slice/crio-77983431df2cfd3d39d1ad779c460510cca12c147bee434a3d2ff80097d9563c WatchSource:0}: Error finding container 77983431df2cfd3d39d1ad779c460510cca12c147bee434a3d2ff80097d9563c: Status 404 returned error can't find the container with id 77983431df2cfd3d39d1ad779c460510cca12c147bee434a3d2ff80097d9563c
Oct 30 22:53:48.023652 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:48.023607    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72" event={"ID":"a5e43d4f-b1f5-4502-bae8-827e24270207","Type":"ContainerStarted","Data":"77983431df2cfd3d39d1ad779c460510cca12c147bee434a3d2ff80097d9563c"}
Oct 30 22:53:48.033515 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:48.033483    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:48.291264 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:48.291234    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"]
Oct 30 22:53:48.293025 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:53:48.292988    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf907df_42f2_4c00_ba1a_60405bdfc3c7.slice/crio-b19abb71b90632d188a4c8c3cbf9ed8fa830f535c695a096787a9064c0011591 WatchSource:0}: Error finding container b19abb71b90632d188a4c8c3cbf9ed8fa830f535c695a096787a9064c0011591: Status 404 returned error can't find the container with id b19abb71b90632d188a4c8c3cbf9ed8fa830f535c695a096787a9064c0011591
Oct 30 22:53:49.033307 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:49.033262    5503 generic.go:356] "Generic (PLEG): container finished" podID="a5e43d4f-b1f5-4502-bae8-827e24270207" containerID="729535a515da8a220ba43e8047629b074dfe14798c3ee753c38d32fa50f1b9a9" exitCode=0
Oct 30 22:53:49.033505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:49.033390    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72" event={"ID":"a5e43d4f-b1f5-4502-bae8-827e24270207","Type":"ContainerDied","Data":"729535a515da8a220ba43e8047629b074dfe14798c3ee753c38d32fa50f1b9a9"}
Oct 30 22:53:49.035338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:49.035297    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx" event={"ID":"ebf907df-42f2-4c00-ba1a-60405bdfc3c7","Type":"ContainerStarted","Data":"c64f8764e43f9862d7cb31a9e2a113ee4ed0d6b9c6dfc16f43f84df246822e50"}
Oct 30 22:53:49.035478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:49.035348    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx" event={"ID":"ebf907df-42f2-4c00-ba1a-60405bdfc3c7","Type":"ContainerStarted","Data":"b19abb71b90632d188a4c8c3cbf9ed8fa830f535c695a096787a9064c0011591"}
Oct 30 22:53:49.257133 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:49.257079    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx" podStartSLOduration=2.257066138 podStartE2EDuration="2.257066138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:49.255479724 +0000 UTC m=+4005.590056495" watchObservedRunningTime="2026-10-30 22:53:49.257066138 +0000 UTC m=+4005.591642907"
Oct 30 22:53:50.044116 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.044080    5503 generic.go:356] "Generic (PLEG): container finished" podID="ebf907df-42f2-4c00-ba1a-60405bdfc3c7" containerID="c64f8764e43f9862d7cb31a9e2a113ee4ed0d6b9c6dfc16f43f84df246822e50" exitCode=0
Oct 30 22:53:50.044562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.044165    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx" event={"ID":"ebf907df-42f2-4c00-ba1a-60405bdfc3c7","Type":"ContainerDied","Data":"c64f8764e43f9862d7cb31a9e2a113ee4ed0d6b9c6dfc16f43f84df246822e50"}
Oct 30 22:53:50.197363 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.197334    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:50.309261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.309171    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-openshift-ca\") pod \"a5e43d4f-b1f5-4502-bae8-827e24270207\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") "
Oct 30 22:53:50.309261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.309216    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e43d4f-b1f5-4502-bae8-827e24270207-kube-api-access-zwb4n\" (UniqueName: \"kubernetes.io/projected/a5e43d4f-b1f5-4502-bae8-827e24270207-kube-api-access-zwb4n\") pod \"a5e43d4f-b1f5-4502-bae8-827e24270207\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") "
Oct 30 22:53:50.309530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.309324    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-xks-ca\") pod \"a5e43d4f-b1f5-4502-bae8-827e24270207\" (UID: \"a5e43d4f-b1f5-4502-bae8-827e24270207\") "
Oct 30 22:53:50.309621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.309589    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-openshift-ca" pod "a5e43d4f-b1f5-4502-bae8-827e24270207" (UID: "a5e43d4f-b1f5-4502-bae8-827e24270207"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:50.309765 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.309747    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:50.311924 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.311897    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e43d4f-b1f5-4502-bae8-827e24270207-kube-api-access-zwb4n" pod "a5e43d4f-b1f5-4502-bae8-827e24270207" (UID: "a5e43d4f-b1f5-4502-bae8-827e24270207"). InnerVolumeSpecName "kube-api-access-zwb4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:50.312110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.311907    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-xks-ca" pod "a5e43d4f-b1f5-4502-bae8-827e24270207" (UID: "a5e43d4f-b1f5-4502-bae8-827e24270207"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:50.411026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.410987    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwb4n\" (UniqueName: \"kubernetes.io/projected/a5e43d4f-b1f5-4502-bae8-827e24270207-kube-api-access-zwb4n\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:50.411026 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:50.411021    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e43d4f-b1f5-4502-bae8-827e24270207-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:51.051570 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.051541    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"
Oct 30 22:53:51.052041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.051570    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72" event={"ID":"a5e43d4f-b1f5-4502-bae8-827e24270207","Type":"ContainerDied","Data":"77983431df2cfd3d39d1ad779c460510cca12c147bee434a3d2ff80097d9563c"}
Oct 30 22:53:51.052041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.051622    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77983431df2cfd3d39d1ad779c460510cca12c147bee434a3d2ff80097d9563c"
Oct 30 22:53:51.196499 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.196472    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:51.218338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.218313    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-xks-ca\") pod \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") "
Oct 30 22:53:51.218477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.218394    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-openshift-ca\") pod \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") "
Oct 30 22:53:51.218477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.218421    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-kube-api-access-v5vx9\" (UniqueName: \"kubernetes.io/projected/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-kube-api-access-v5vx9\") pod \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\" (UID: \"ebf907df-42f2-4c00-ba1a-60405bdfc3c7\") "
Oct 30 22:53:51.218760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.218738    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-openshift-ca" pod "ebf907df-42f2-4c00-ba1a-60405bdfc3c7" (UID: "ebf907df-42f2-4c00-ba1a-60405bdfc3c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:51.220376 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.220351    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-xks-ca" pod "ebf907df-42f2-4c00-ba1a-60405bdfc3c7" (UID: "ebf907df-42f2-4c00-ba1a-60405bdfc3c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:51.220472 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.220412    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-kube-api-access-v5vx9" pod "ebf907df-42f2-4c00-ba1a-60405bdfc3c7" (UID: "ebf907df-42f2-4c00-ba1a-60405bdfc3c7"). InnerVolumeSpecName "kube-api-access-v5vx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:51.319315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.319227    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:51.319315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.319265    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:51.319315 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:51.319277    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5vx9\" (UniqueName: \"kubernetes.io/projected/ebf907df-42f2-4c00-ba1a-60405bdfc3c7-kube-api-access-v5vx9\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:52.059671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:52.059568    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"
Oct 30 22:53:52.059671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:52.059568    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx" event={"ID":"ebf907df-42f2-4c00-ba1a-60405bdfc3c7","Type":"ContainerDied","Data":"b19abb71b90632d188a4c8c3cbf9ed8fa830f535c695a096787a9064c0011591"}
Oct 30 22:53:52.060126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:52.059697    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19abb71b90632d188a4c8c3cbf9ed8fa830f535c695a096787a9064c0011591"
Oct 30 22:53:52.755578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:52.755546    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:52.799579 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:52.799546    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"]
Oct 30 22:53:52.901417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:52.901382    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"]
Oct 30 22:53:52.980104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:52.980073    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:53.173445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:53.173408    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:53.324798 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:53.324765    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:57.326525 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:57.326493    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:53:57.800181 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:57.800127    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" podUID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" containerName="payload-processing" containerID="cri-o://3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227" gracePeriod=25
Oct 30 22:53:57.901703 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:57.901657    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" podUID="d9e53a02-7158-4bc0-91b3-177c5e587222" containerName="payload-pre-processing" containerID="cri-o://d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b" gracePeriod=25
Oct 30 22:53:58.098786 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:53:58.098697    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:53:58.480624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:58.480585    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" podUID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:53:58.480624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:53:58.480606    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" podUID="d9e53a02-7158-4bc0-91b3-177c5e587222" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:54:01.290457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:01.290401    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:02.755931 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:02.755889    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:03.038829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:03.038736    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:03.197395 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:03.197364    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:03.299878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:03.299744    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"]
Oct 30 22:54:03.346490 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:03.346437    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-435923-sqzsn"]
Oct 30 22:54:03.703459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:03.703429    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:03.788956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:03.788925    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad2746d-9ef0-46de-ac7b-7a6c450a7965" path="/var/lib/kubelet/pods/aad2746d-9ef0-46de-ac7b-7a6c450a7965/volumes"
Oct 30 22:54:06.175976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:06.175944    5503 scope.go:122] "RemoveContainer" containerID="73af409bd692502a6aa4b75d60e73addfd203dd1ee63886420aee95d40fc19a4"
Oct 30 22:54:07.916982 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:07.916943    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:08.480672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:08.480619    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" podUID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:54:08.480672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:08.480619    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" podUID="d9e53a02-7158-4bc0-91b3-177c5e587222" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:54:08.676267 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:08.676234    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:09.824002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:09.823969    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:09.979602 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:09.979566    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:10.180595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:10.180559    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"]
Oct 30 22:54:10.180809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:10.180785    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy" containerID="cri-o://aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339" gracePeriod=30
Oct 30 22:54:10.193947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:10.193916    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:10.682297 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:10.682263    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:11.190141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.190107    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"]
Oct 30 22:54:11.190591 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.190578    5503 cpu_manager.go:414] "containerMap: removing container" podUID="ebf907df-42f2-4c00-ba1a-60405bdfc3c7" containerName="revoke-keys"
Oct 30 22:54:11.190655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.190591    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebf907df-42f2-4c00-ba1a-60405bdfc3c7" containerName="revoke-keys"
Oct 30 22:54:11.190655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.190619    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a5e43d4f-b1f5-4502-bae8-827e24270207" containerName="revoke-keys"
Oct 30 22:54:11.190655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.190624    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e43d4f-b1f5-4502-bae8-827e24270207" containerName="revoke-keys"
Oct 30 22:54:11.190760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.190708    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e43d4f-b1f5-4502-bae8-827e24270207" containerName="revoke-keys"
Oct 30 22:54:11.190760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.190722    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebf907df-42f2-4c00-ba1a-60405bdfc3c7" containerName="revoke-keys"
Oct 30 22:54:11.193947 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.193931    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.249352 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.249319    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"]
Oct 30 22:54:11.249832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.249787    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psbxp\" (UniqueName: \"kubernetes.io/projected/135d5792-a394-4f6a-8fdf-7596b4fef503-kube-api-access-psbxp\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.250019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.249856    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.250090 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.250021    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.293117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.293090    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:11.293333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.293200    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:11.293333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.293200    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:54:11.293333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.293203    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:11.351299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.351264    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psbxp\" (UniqueName: \"kubernetes.io/projected/135d5792-a394-4f6a-8fdf-7596b4fef503-kube-api-access-psbxp\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.351478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.351327    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.351478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.351400    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.352033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.352014    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.353788 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.353760    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.410655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.410619    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psbxp\" (UniqueName: \"kubernetes.io/projected/135d5792-a394-4f6a-8fdf-7596b4fef503-kube-api-access-psbxp\") pod \"maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.506655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.506548    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:11.875819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:11.875793    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"]
Oct 30 22:54:11.877535 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:54:11.877507    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135d5792_a394_4f6a_8fdf_7596b4fef503.slice/crio-5a9559416e68d54e40a0259e4f70441b4b18c05a17c15952f867bbf51d430c08 WatchSource:0}: Error finding container 5a9559416e68d54e40a0259e4f70441b4b18c05a17c15952f867bbf51d430c08: Status 404 returned error can't find the container with id 5a9559416e68d54e40a0259e4f70441b4b18c05a17c15952f867bbf51d430c08
Oct 30 22:54:12.184815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:12.184776    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq" event={"ID":"135d5792-a394-4f6a-8fdf-7596b4fef503","Type":"ContainerStarted","Data":"ff2222fff03be71d89f3c581bafdb5623ee7ca9195cec058a13e7678f078be9c"}
Oct 30 22:54:12.185023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:12.184834    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq" event={"ID":"135d5792-a394-4f6a-8fdf-7596b4fef503","Type":"ContainerStarted","Data":"5a9559416e68d54e40a0259e4f70441b4b18c05a17c15952f867bbf51d430c08"}
Oct 30 22:54:12.296313 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:12.296255    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq" podStartSLOduration=1.29624089 podStartE2EDuration="1.29624089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:12.29427005 +0000 UTC m=+4028.628846821" watchObservedRunningTime="2026-10-30 22:54:12.29624089 +0000 UTC m=+4028.630817709"
Oct 30 22:54:12.647781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:12.647745    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:13.191831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:13.191794    5503 generic.go:356] "Generic (PLEG): container finished" podID="135d5792-a394-4f6a-8fdf-7596b4fef503" containerID="ff2222fff03be71d89f3c581bafdb5623ee7ca9195cec058a13e7678f078be9c" exitCode=0
Oct 30 22:54:13.192017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:13.191878    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq" event={"ID":"135d5792-a394-4f6a-8fdf-7596b4fef503","Type":"ContainerDied","Data":"ff2222fff03be71d89f3c581bafdb5623ee7ca9195cec058a13e7678f078be9c"}
Oct 30 22:54:14.342423 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.342397    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:14.486337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.486256    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135d5792-a394-4f6a-8fdf-7596b4fef503-kube-api-access-psbxp\" (UniqueName: \"kubernetes.io/projected/135d5792-a394-4f6a-8fdf-7596b4fef503-kube-api-access-psbxp\") pod \"135d5792-a394-4f6a-8fdf-7596b4fef503\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") "
Oct 30 22:54:14.486337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.486309    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-xks-ca\") pod \"135d5792-a394-4f6a-8fdf-7596b4fef503\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") "
Oct 30 22:54:14.486546 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.486375    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-openshift-ca\") pod \"135d5792-a394-4f6a-8fdf-7596b4fef503\" (UID: \"135d5792-a394-4f6a-8fdf-7596b4fef503\") "
Oct 30 22:54:14.486738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.486711    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-openshift-ca" pod "135d5792-a394-4f6a-8fdf-7596b4fef503" (UID: "135d5792-a394-4f6a-8fdf-7596b4fef503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:14.486829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.486812    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:14.488460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.488434    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-xks-ca" pod "135d5792-a394-4f6a-8fdf-7596b4fef503" (UID: "135d5792-a394-4f6a-8fdf-7596b4fef503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:14.488599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.488572    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135d5792-a394-4f6a-8fdf-7596b4fef503-kube-api-access-psbxp" pod "135d5792-a394-4f6a-8fdf-7596b4fef503" (UID: "135d5792-a394-4f6a-8fdf-7596b4fef503"). InnerVolumeSpecName "kube-api-access-psbxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:14.588087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.588052    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psbxp\" (UniqueName: \"kubernetes.io/projected/135d5792-a394-4f6a-8fdf-7596b4fef503-kube-api-access-psbxp\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:14.588087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:14.588081    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d5792-a394-4f6a-8fdf-7596b4fef503-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.206186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.206158    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"
Oct 30 22:54:15.206385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.206158    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq" event={"ID":"135d5792-a394-4f6a-8fdf-7596b4fef503","Type":"ContainerDied","Data":"5a9559416e68d54e40a0259e4f70441b4b18c05a17c15952f867bbf51d430c08"}
Oct 30 22:54:15.206385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.206360    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a9559416e68d54e40a0259e4f70441b4b18c05a17c15952f867bbf51d430c08"
Oct 30 22:54:15.442184 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.442159    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:54:15.599650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.599543    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-podinfo\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.599835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.599626    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-token\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-token\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.599835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.599711    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1891cfd-4b00-4edc-8303-226a3d4499fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1891cfd-4b00-4edc-8303-226a3d4499fd-istiod-ca-cert\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.599835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.599752    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-envoy\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.599835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.599802    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-data\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.600069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.599845    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-socket\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.600069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.599902    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-kube-api-access-djnsh\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-kube-api-access-djnsh\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.600069 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600029    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-certs\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.600225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600067    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-credential-socket\") pod \"a1891cfd-4b00-4edc-8303-226a3d4499fd\" (UID: \"a1891cfd-4b00-4edc-8303-226a3d4499fd\") "
Oct 30 22:54:15.600225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600102    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-data" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:15.600225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600156    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-socket" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:15.600225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600213    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1891cfd-4b00-4edc-8303-226a3d4499fd-istiod-ca-cert" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:15.600457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600392    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-certs" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:15.600521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600451    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-credential-socket" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:15.600592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600573    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.600676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600599    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.600676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600614    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1891cfd-4b00-4edc-8303-226a3d4499fd-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.600676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600628    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.600676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.600659    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.601996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.601973    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-podinfo" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:54:15.602433 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.602411    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-kube-api-access-djnsh" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "kube-api-access-djnsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:15.602433 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.602404    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-envoy" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:15.602578 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.602434    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-token" pod "a1891cfd-4b00-4edc-8303-226a3d4499fd" (UID: "a1891cfd-4b00-4edc-8303-226a3d4499fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:15.631437 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.631403    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"]
Oct 30 22:54:15.637066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.637038    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"]
Oct 30 22:54:15.645095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.645068    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"]
Oct 30 22:54:15.655550 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.655525    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"]
Oct 30 22:54:15.702079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.702047    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnsh\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-kube-api-access-djnsh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.702257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.702097    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.702257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.702108    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.702257 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.702117    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1891cfd-4b00-4edc-8303-226a3d4499fd-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:15.831468 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:15.831435    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:16.213562 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.213528    5503 generic.go:356] "Generic (PLEG): container finished" podID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerID="aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339" exitCode=0
Oct 30 22:54:16.213769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.213591    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"
Oct 30 22:54:16.213769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.213611    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" event={"ID":"a1891cfd-4b00-4edc-8303-226a3d4499fd","Type":"ContainerDied","Data":"aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339"}
Oct 30 22:54:16.213769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.213693    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm" event={"ID":"a1891cfd-4b00-4edc-8303-226a3d4499fd","Type":"ContainerDied","Data":"b7f0338e48847029c39048b282a65747ebad0e79fdd3697efc760442529bc3f8"}
Oct 30 22:54:16.213769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.213720    5503 scope.go:122] "RemoveContainer" containerID="aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339"
Oct 30 22:54:16.223744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.223726    5503 scope.go:122] "RemoveContainer" containerID="c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb"
Oct 30 22:54:16.232608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.232591    5503 scope.go:122] "RemoveContainer" containerID="aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339"
Oct 30 22:54:16.232890 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:16.232870    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339\": container with ID starting with aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339 not found: ID does not exist" containerID="aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339"
Oct 30 22:54:16.232973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.232898    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339"} err="failed to get container status \"aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339\": rpc error: code = NotFound desc = could not find container \"aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339\": container with ID starting with aa790c5d64998d6fc782f44f6911281d16bc630f4bc2dcc32513a3ec3cb5d339 not found: ID does not exist"
Oct 30 22:54:16.232973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.232916    5503 scope.go:122] "RemoveContainer" containerID="c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb"
Oct 30 22:54:16.233151 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:16.233131    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb\": container with ID starting with c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb not found: ID does not exist" containerID="c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb"
Oct 30 22:54:16.233216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.233156    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb"} err="failed to get container status \"c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb\": rpc error: code = NotFound desc = could not find container \"c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb\": container with ID starting with c8df9bc32d0f0eb52d94b7d52c4506aa9e220e3a9edb4590132de7c34dceddcb not found: ID does not exist"
Oct 30 22:54:16.372817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.372783    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"]
Oct 30 22:54:16.377537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:16.377513    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-435923-openshift-default-658c6d685f-nl6hm"]
Oct 30 22:54:17.429817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:17.429787    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:17.652349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:17.652314    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:54:17.791912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:17.791827    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" path="/var/lib/kubelet/pods/a1891cfd-4b00-4edc-8303-226a3d4499fd/volumes"
Oct 30 22:54:18.049541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.049476    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:54:18.130354 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.130322    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6541ed5e-f464-4091-93aa-b8270ff2e4dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6541ed5e-f464-4091-93aa-b8270ff2e4dc-plugins-config-volume\") pod \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") "
Oct 30 22:54:18.130515 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.130449    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6541ed5e-f464-4091-93aa-b8270ff2e4dc-kube-api-access-zppsz\" (UniqueName: \"kubernetes.io/projected/6541ed5e-f464-4091-93aa-b8270ff2e4dc-kube-api-access-zppsz\") pod \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\" (UID: \"6541ed5e-f464-4091-93aa-b8270ff2e4dc\") "
Oct 30 22:54:18.130678 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.130652    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6541ed5e-f464-4091-93aa-b8270ff2e4dc-plugins-config-volume" pod "6541ed5e-f464-4091-93aa-b8270ff2e4dc" (UID: "6541ed5e-f464-4091-93aa-b8270ff2e4dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:18.130768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.130752    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6541ed5e-f464-4091-93aa-b8270ff2e4dc-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:18.132374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.132352    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6541ed5e-f464-4091-93aa-b8270ff2e4dc-kube-api-access-zppsz" pod "6541ed5e-f464-4091-93aa-b8270ff2e4dc" (UID: "6541ed5e-f464-4091-93aa-b8270ff2e4dc"). InnerVolumeSpecName "kube-api-access-zppsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:18.145225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.145204    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:54:18.229150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.229120    5503 generic.go:356] "Generic (PLEG): container finished" podID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" containerID="3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227" exitCode=0
Oct 30 22:54:18.229338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.229173    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"
Oct 30 22:54:18.229338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.229208    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" event={"ID":"6541ed5e-f464-4091-93aa-b8270ff2e4dc","Type":"ContainerDied","Data":"3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227"}
Oct 30 22:54:18.229338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.229262    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng" event={"ID":"6541ed5e-f464-4091-93aa-b8270ff2e4dc","Type":"ContainerDied","Data":"0f48a01484892ea4981f7d87efda13dc3eeb5c02218b5c41f92ae055f5b65c44"}
Oct 30 22:54:18.229338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.229293    5503 scope.go:122] "RemoveContainer" containerID="3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227"
Oct 30 22:54:18.229566 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.229389    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:18.230873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.230848    5503 generic.go:356] "Generic (PLEG): container finished" podID="d9e53a02-7158-4bc0-91b3-177c5e587222" containerID="d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b" exitCode=0
Oct 30 22:54:18.231009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.230885    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"
Oct 30 22:54:18.231009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.230936    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" event={"ID":"d9e53a02-7158-4bc0-91b3-177c5e587222","Type":"ContainerDied","Data":"d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b"}
Oct 30 22:54:18.231009 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.230988    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9" event={"ID":"d9e53a02-7158-4bc0-91b3-177c5e587222","Type":"ContainerDied","Data":"891a036eae936138f51a95744e8bcf0c818e3f511647147a4d17e3fcf53fd6f0"}
Oct 30 22:54:18.231166 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.231151    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e53a02-7158-4bc0-91b3-177c5e587222-kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/d9e53a02-7158-4bc0-91b3-177c5e587222-kube-api-access-whmzd\") pod \"d9e53a02-7158-4bc0-91b3-177c5e587222\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") "
Oct 30 22:54:18.231202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.231194    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e53a02-7158-4bc0-91b3-177c5e587222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e53a02-7158-4bc0-91b3-177c5e587222-plugins-config-volume\") pod \"d9e53a02-7158-4bc0-91b3-177c5e587222\" (UID: \"d9e53a02-7158-4bc0-91b3-177c5e587222\") "
Oct 30 22:54:18.231601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.231582    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zppsz\" (UniqueName: \"kubernetes.io/projected/6541ed5e-f464-4091-93aa-b8270ff2e4dc-kube-api-access-zppsz\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:18.231736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.231673    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e53a02-7158-4bc0-91b3-177c5e587222-plugins-config-volume" pod "d9e53a02-7158-4bc0-91b3-177c5e587222" (UID: "d9e53a02-7158-4bc0-91b3-177c5e587222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:18.233349 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.233328    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e53a02-7158-4bc0-91b3-177c5e587222-kube-api-access-whmzd" pod "d9e53a02-7158-4bc0-91b3-177c5e587222" (UID: "d9e53a02-7158-4bc0-91b3-177c5e587222"). InnerVolumeSpecName "kube-api-access-whmzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:18.239309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.239288    5503 scope.go:122] "RemoveContainer" containerID="3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227"
Oct 30 22:54:18.239581 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:18.239559    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227\": container with ID starting with 3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227 not found: ID does not exist" containerID="3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227"
Oct 30 22:54:18.239656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.239587    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227"} err="failed to get container status \"3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227\": rpc error: code = NotFound desc = could not find container \"3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227\": container with ID starting with 3e667bf0f9b7f34556a1098c8c606723b0fcf8952d176c34bc5815b11e8f0227 not found: ID does not exist"
Oct 30 22:54:18.239656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.239603    5503 scope.go:122] "RemoveContainer" containerID="d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b"
Oct 30 22:54:18.248873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.248854    5503 scope.go:122] "RemoveContainer" containerID="d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b"
Oct 30 22:54:18.249138 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:18.249113    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b\": container with ID starting with d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b not found: ID does not exist" containerID="d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b"
Oct 30 22:54:18.249189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.249146    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b"} err="failed to get container status \"d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b\": rpc error: code = NotFound desc = could not find container \"d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b\": container with ID starting with d56fd7e6db09102d17a79225d62df4cdfd4333d3d92114eb1057fdd6b577709b not found: ID does not exist"
Oct 30 22:54:18.332953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.332878    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whmzd\" (UniqueName: \"kubernetes.io/projected/d9e53a02-7158-4bc0-91b3-177c5e587222-kube-api-access-whmzd\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:18.332953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.332903    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e53a02-7158-4bc0-91b3-177c5e587222-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:18.379552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.379519    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:18.431082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.431048    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"]
Oct 30 22:54:18.435607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.435578    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-1b5075-6b8f5b6d97-d8cng"]
Oct 30 22:54:18.648584 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.648497    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"]
Oct 30 22:54:18.651792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:18.651764    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1b5075-79b4598787-57xh9"]
Oct 30 22:54:19.789215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:19.789179    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" path="/var/lib/kubelet/pods/6541ed5e-f464-4091-93aa-b8270ff2e4dc/volumes"
Oct 30 22:54:19.789799 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:19.789778    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e53a02-7158-4bc0-91b3-177c5e587222" path="/var/lib/kubelet/pods/d9e53a02-7158-4bc0-91b3-177c5e587222/volumes"
Oct 30 22:54:22.108384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:22.108347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:23.117155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:23.117120    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:23.270739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:23.270699    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:26.748879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:26.748841    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:28.098780 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:28.098746    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:54:30.653044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.653001    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" podUID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" containerName="main" containerID="cri-o://24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717" gracePeriod=45
Oct 30 22:54:30.658113 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.658042    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv" podUID="d07b7d4d-0016-477f-8edc-47db1175e93b" containerName="main" containerID="cri-o://775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed" gracePeriod=45
Oct 30 22:54:30.665310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.665252    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main" containerID="cri-o://b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a" gracePeriod=45
Oct 30 22:54:30.674603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.674568    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main" containerID="cri-o://679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf" gracePeriod=45
Oct 30 22:54:30.687096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687065    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"]
Oct 30 22:54:30.687544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687531    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy"
Oct 30 22:54:30.687595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687546    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy"
Oct 30 22:54:30.687595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687576    5503 cpu_manager.go:414] "containerMap: removing container" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy"
Oct 30 22:54:30.687595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687583    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy"
Oct 30 22:54:30.687595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687595    5503 cpu_manager.go:414] "containerMap: removing container" podUID="135d5792-a394-4f6a-8fdf-7596b4fef503" containerName="revoke-keys"
Oct 30 22:54:30.687747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687600    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135d5792-a394-4f6a-8fdf-7596b4fef503" containerName="revoke-keys"
Oct 30 22:54:30.687747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687616    5503 cpu_manager.go:414] "containerMap: removing container" podUID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" containerName="payload-processing"
Oct 30 22:54:30.687747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687621    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" containerName="payload-processing"
Oct 30 22:54:30.687747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687629    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d9e53a02-7158-4bc0-91b3-177c5e587222" containerName="payload-pre-processing"
Oct 30 22:54:30.687747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687648    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e53a02-7158-4bc0-91b3-177c5e587222" containerName="payload-pre-processing"
Oct 30 22:54:30.687747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687729    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy"
Oct 30 22:54:30.687747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687739    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="135d5792-a394-4f6a-8fdf-7596b4fef503" containerName="revoke-keys"
Oct 30 22:54:30.687962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687751    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6541ed5e-f464-4091-93aa-b8270ff2e4dc" containerName="payload-processing"
Oct 30 22:54:30.687962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.687761    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e53a02-7158-4bc0-91b3-177c5e587222" containerName="payload-pre-processing"
Oct 30 22:54:30.691175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.691156    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.742829 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.742798    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"]
Oct 30 22:54:30.743368 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.743352    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1891cfd-4b00-4edc-8303-226a3d4499fd" containerName="istio-proxy"
Oct 30 22:54:30.746674 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.746647    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.753164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.753130    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7vr\" (UniqueName: \"kubernetes.io/projected/999c1eff-d672-4255-92f5-eb3076411333-kube-api-access-kg7vr\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.753313 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.753256    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c1eff-d672-4255-92f5-eb3076411333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.753388 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.753346    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.753474 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.753450    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gb4\" (UniqueName: \"kubernetes.io/projected/e59b1fa2-41ab-459f-b139-3cfc31c7d521-kube-api-access-z5gb4\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.753535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.753504    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.753593 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.753540    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c1eff-d672-4255-92f5-eb3076411333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.770683 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.770625    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" podUID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 30 22:54:30.787412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.787299    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:30.787412 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.787331    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:30.791253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.791230    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:54:30.791447 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.791327    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:30.792063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.792037    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"]
Oct 30 22:54:30.801746 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.801663    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"]
Oct 30 22:54:30.855253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.854395    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gb4\" (UniqueName: \"kubernetes.io/projected/e59b1fa2-41ab-459f-b139-3cfc31c7d521-kube-api-access-z5gb4\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.855253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.854452    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.855253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.854484    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c1eff-d672-4255-92f5-eb3076411333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.855253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.854530    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7vr\" (UniqueName: \"kubernetes.io/projected/999c1eff-d672-4255-92f5-eb3076411333-kube-api-access-kg7vr\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.855253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.854596    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c1eff-d672-4255-92f5-eb3076411333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.855253 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.854682    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.858702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.856516    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c1eff-d672-4255-92f5-eb3076411333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.858702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.857453    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.859348 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.859295    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.862117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.862090    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c1eff-d672-4255-92f5-eb3076411333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.904152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.903985    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:30.918865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.918810    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7vr\" (UniqueName: \"kubernetes.io/projected/999c1eff-d672-4255-92f5-eb3076411333-kube-api-access-kg7vr\") pod \"maas-api-revoke-sub-85e70155-c4lk8\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:30.919050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.918902    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gb4\" (UniqueName: \"kubernetes.io/projected/e59b1fa2-41ab-459f-b139-3cfc31c7d521-kube-api-access-z5gb4\") pod \"maas-api-revoke-sub-d90c6b12-kgwcs\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:30.994314 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:30.993968    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:54:31.031085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.030797    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:54:31.043129 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.043105    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:54:31.047073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.047052    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:54:31.057494 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057464    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tls-certs\") pod \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") "
Oct 30 22:54:31.057670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057538    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9114be59-0719-48bc-9695-933cf4606b43-tls-certs\" (UniqueName: \"kubernetes.io/secret/9114be59-0719-48bc-9695-933cf4606b43-tls-certs\") pod \"9114be59-0719-48bc-9695-933cf4606b43\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") "
Oct 30 22:54:31.057670 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057628    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0dec83-8177-4bab-82dc-d62f1d5a7258-kube-api-access-8mj4x\" (UniqueName: \"kubernetes.io/projected/fc0dec83-8177-4bab-82dc-d62f1d5a7258-kube-api-access-8mj4x\") pod \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") "
Oct 30 22:54:31.057822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057714    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-tmp-dir\") pod \"d07b7d4d-0016-477f-8edc-47db1175e93b\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") "
Oct 30 22:54:31.057822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057769    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-dshm\") pod \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") "
Oct 30 22:54:31.057822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057797    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-model-cache\") pod \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") "
Oct 30 22:54:31.057972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057828    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07b7d4d-0016-477f-8edc-47db1175e93b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d07b7d4d-0016-477f-8edc-47db1175e93b-tls-certs\") pod \"d07b7d4d-0016-477f-8edc-47db1175e93b\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") "
Oct 30 22:54:31.057972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057873    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-home\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-home\") pod \"9114be59-0719-48bc-9695-933cf4606b43\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") "
Oct 30 22:54:31.057972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057911    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-home\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-home\") pod \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") "
Oct 30 22:54:31.057972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057941    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9114be59-0719-48bc-9695-933cf4606b43-kube-api-access-r5jq4\" (UniqueName: \"kubernetes.io/projected/9114be59-0719-48bc-9695-933cf4606b43-kube-api-access-r5jq4\") pod \"9114be59-0719-48bc-9695-933cf4606b43\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") "
Oct 30 22:54:31.058195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.057971    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-home\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-home\") pod \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") "
Oct 30 22:54:31.058195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058014    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-tmp-dir" pod "d07b7d4d-0016-477f-8edc-47db1175e93b" (UID: "d07b7d4d-0016-477f-8edc-47db1175e93b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.058195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058077    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-tmp-dir\") pod \"9114be59-0719-48bc-9695-933cf4606b43\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") "
Oct 30 22:54:31.058195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058119    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07b7d4d-0016-477f-8edc-47db1175e93b-kube-api-access-f2k25\" (UniqueName: \"kubernetes.io/projected/d07b7d4d-0016-477f-8edc-47db1175e93b-kube-api-access-f2k25\") pod \"d07b7d4d-0016-477f-8edc-47db1175e93b\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") "
Oct 30 22:54:31.058195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058139    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-model-cache" pod "fc0dec83-8177-4bab-82dc-d62f1d5a7258" (UID: "fc0dec83-8177-4bab-82dc-d62f1d5a7258"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.058195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058173    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-model-cache\") pod \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058228    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-model-cache\") pod \"d07b7d4d-0016-477f-8edc-47db1175e93b\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058237    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-home" pod "fc0dec83-8177-4bab-82dc-d62f1d5a7258" (UID: "fc0dec83-8177-4bab-82dc-d62f1d5a7258"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058312    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-dshm\") pod \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058343    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-home\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-home\") pod \"d07b7d4d-0016-477f-8edc-47db1175e93b\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058349    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-home" pod "9114be59-0719-48bc-9695-933cf4606b43" (UID: "9114be59-0719-48bc-9695-933cf4606b43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058387    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-dshm\") pod \"9114be59-0719-48bc-9695-933cf4606b43\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058423    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tmp-dir\") pod \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058453    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-kube-api-access-wbp7v\" (UniqueName: \"kubernetes.io/projected/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-kube-api-access-wbp7v\") pod \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\" (UID: \"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058472    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-model-cache" pod "d07b7d4d-0016-477f-8edc-47db1175e93b" (UID: "d07b7d4d-0016-477f-8edc-47db1175e93b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058500    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tmp-dir\") pod \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058540    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tls-certs\") pod \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\" (UID: \"fc0dec83-8177-4bab-82dc-d62f1d5a7258\") "
Oct 30 22:54:31.058623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058576    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-dshm\") pod \"d07b7d4d-0016-477f-8edc-47db1175e93b\" (UID: \"d07b7d4d-0016-477f-8edc-47db1175e93b\") "
Oct 30 22:54:31.059312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.058671    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-tmp-dir" pod "9114be59-0719-48bc-9695-933cf4606b43" (UID: "9114be59-0719-48bc-9695-933cf4606b43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.059312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.059202    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.059312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.059226    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.059312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.059240    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.059312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.059254    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.059312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.059267    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.059312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.059279    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.060503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.060477    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-home" pod "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" (UID: "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.060621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.060538    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tmp-dir" pod "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" (UID: "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.060621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.060576    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-model-cache" pod "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" (UID: "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.060831 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.060624    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07b7d4d-0016-477f-8edc-47db1175e93b-tls-certs" pod "d07b7d4d-0016-477f-8edc-47db1175e93b" (UID: "d07b7d4d-0016-477f-8edc-47db1175e93b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:31.061084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.061061    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-home" pod "d07b7d4d-0016-477f-8edc-47db1175e93b" (UID: "d07b7d4d-0016-477f-8edc-47db1175e93b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.061249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.061113    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07b7d4d-0016-477f-8edc-47db1175e93b-kube-api-access-f2k25" pod "d07b7d4d-0016-477f-8edc-47db1175e93b" (UID: "d07b7d4d-0016-477f-8edc-47db1175e93b"). InnerVolumeSpecName "kube-api-access-f2k25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:31.061337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.061258    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tmp-dir" pod "fc0dec83-8177-4bab-82dc-d62f1d5a7258" (UID: "fc0dec83-8177-4bab-82dc-d62f1d5a7258"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.061663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.061599    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9114be59-0719-48bc-9695-933cf4606b43-kube-api-access-r5jq4" pod "9114be59-0719-48bc-9695-933cf4606b43" (UID: "9114be59-0719-48bc-9695-933cf4606b43"). InnerVolumeSpecName "kube-api-access-r5jq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:31.062156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.062074    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9114be59-0719-48bc-9695-933cf4606b43-tls-certs" pod "9114be59-0719-48bc-9695-933cf4606b43" (UID: "9114be59-0719-48bc-9695-933cf4606b43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:31.062714 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.062688    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-kube-api-access-wbp7v" pod "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" (UID: "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4"). InnerVolumeSpecName "kube-api-access-wbp7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:31.062825 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.062693    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tls-certs" pod "fc0dec83-8177-4bab-82dc-d62f1d5a7258" (UID: "fc0dec83-8177-4bab-82dc-d62f1d5a7258"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:31.063122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.063084    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-dshm" pod "fc0dec83-8177-4bab-82dc-d62f1d5a7258" (UID: "fc0dec83-8177-4bab-82dc-d62f1d5a7258"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.063122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.063098    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-dshm" pod "d07b7d4d-0016-477f-8edc-47db1175e93b" (UID: "d07b7d4d-0016-477f-8edc-47db1175e93b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.063592 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.063572    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-dshm" pod "9114be59-0719-48bc-9695-933cf4606b43" (UID: "9114be59-0719-48bc-9695-933cf4606b43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.063707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.063626    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0dec83-8177-4bab-82dc-d62f1d5a7258-kube-api-access-8mj4x" pod "fc0dec83-8177-4bab-82dc-d62f1d5a7258" (UID: "fc0dec83-8177-4bab-82dc-d62f1d5a7258"). InnerVolumeSpecName "kube-api-access-8mj4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:31.063784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.063714    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tls-certs" pod "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" (UID: "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:31.063784 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.063731    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-dshm" pod "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" (UID: "e8c62843-bc04-4d6a-a63e-ca97b65f9fc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.159956 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.159878    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-model-cache\") pod \"9114be59-0719-48bc-9695-933cf4606b43\" (UID: \"9114be59-0719-48bc-9695-933cf4606b43\") "
Oct 30 22:54:31.160139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160127    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160145    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160154    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160162    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160172    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbp7v\" (UniqueName: \"kubernetes.io/projected/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-kube-api-access-wbp7v\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160183    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160194 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160191    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0dec83-8177-4bab-82dc-d62f1d5a7258-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160199    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d07b7d4d-0016-477f-8edc-47db1175e93b-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160121    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-model-cache" pod "9114be59-0719-48bc-9695-933cf4606b43" (UID: "9114be59-0719-48bc-9695-933cf4606b43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160207    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160239    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9114be59-0719-48bc-9695-933cf4606b43-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160251    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mj4x\" (UniqueName: \"kubernetes.io/projected/fc0dec83-8177-4bab-82dc-d62f1d5a7258-kube-api-access-8mj4x\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160262    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0dec83-8177-4bab-82dc-d62f1d5a7258-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160272    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d07b7d4d-0016-477f-8edc-47db1175e93b-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160281    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5jq4\" (UniqueName: \"kubernetes.io/projected/9114be59-0719-48bc-9695-933cf4606b43-kube-api-access-r5jq4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160290    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160299    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2k25\" (UniqueName: \"kubernetes.io/projected/d07b7d4d-0016-477f-8edc-47db1175e93b-kube-api-access-f2k25\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.160383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.160307    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.200884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.200853    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:31.210520 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.210485    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:31.261459 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.261422    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9114be59-0719-48bc-9695-933cf4606b43-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.316619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.316583    5503 generic.go:356] "Generic (PLEG): container finished" podID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" containerID="24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717" exitCode=0
Oct 30 22:54:31.316842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.316664    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"
Oct 30 22:54:31.316842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.316687    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" event={"ID":"fc0dec83-8177-4bab-82dc-d62f1d5a7258","Type":"ContainerDied","Data":"24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717"}
Oct 30 22:54:31.316842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.316735    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74" event={"ID":"fc0dec83-8177-4bab-82dc-d62f1d5a7258","Type":"ContainerDied","Data":"254d4b7f8448de5ddca2c1b9e27262d4ce4f76e9a48e892b9e90feb8c884b247"}
Oct 30 22:54:31.316842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.316761    5503 scope.go:122] "RemoveContainer" containerID="24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717"
Oct 30 22:54:31.318511 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.318489    5503 generic.go:356] "Generic (PLEG): container finished" podID="9114be59-0719-48bc-9695-933cf4606b43" containerID="b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a" exitCode=0
Oct 30 22:54:31.318600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.318534    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"
Oct 30 22:54:31.318600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.318571    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" event={"ID":"9114be59-0719-48bc-9695-933cf4606b43","Type":"ContainerDied","Data":"b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a"}
Oct 30 22:54:31.318730 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.318612    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr" event={"ID":"9114be59-0719-48bc-9695-933cf4606b43","Type":"ContainerDied","Data":"206b59aba099353c36f8f74229762a83817b6ffedb93eb085b863414772296be"}
Oct 30 22:54:31.323776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.320184    5503 generic.go:356] "Generic (PLEG): container finished" podID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerID="679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf" exitCode=0
Oct 30 22:54:31.323776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.320298    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" event={"ID":"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4","Type":"ContainerDied","Data":"679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf"}
Oct 30 22:54:31.323776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.320324    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" event={"ID":"e8c62843-bc04-4d6a-a63e-ca97b65f9fc4","Type":"ContainerDied","Data":"4fb2202ec7247e22160423abad7b3a020aaeddfbfb6b96588608733e91685f4b"}
Oct 30 22:54:31.323776 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.320384    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"
Oct 30 22:54:31.327673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.325321    5503 generic.go:356] "Generic (PLEG): container finished" podID="d07b7d4d-0016-477f-8edc-47db1175e93b" containerID="775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed" exitCode=0
Oct 30 22:54:31.327673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.325439    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv" event={"ID":"d07b7d4d-0016-477f-8edc-47db1175e93b","Type":"ContainerDied","Data":"775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed"}
Oct 30 22:54:31.327673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.325468    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv" event={"ID":"d07b7d4d-0016-477f-8edc-47db1175e93b","Type":"ContainerDied","Data":"2248e5a872c54f630f9463944e820e7eee416be64b281b75f5c40fd57507d77a"}
Oct 30 22:54:31.327673 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.325529    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"
Oct 30 22:54:31.367340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.367311    5503 scope.go:122] "RemoveContainer" containerID="24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717"
Oct 30 22:54:31.374927 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:31.374890    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717\": container with ID starting with 24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717 not found: ID does not exist" containerID="24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717"
Oct 30 22:54:31.375186 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.375148    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717"} err="failed to get container status \"24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717\": rpc error: code = NotFound desc = could not find container \"24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717\": container with ID starting with 24586f184c41f7471a92bd8cac7435097edb567243b7636989c8b773bfdfc717 not found: ID does not exist"
Oct 30 22:54:31.375337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.375320    5503 scope.go:122] "RemoveContainer" containerID="b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a"
Oct 30 22:54:31.390794 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.390760    5503 scope.go:122] "RemoveContainer" containerID="b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a"
Oct 30 22:54:31.391361 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:31.391334    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a\": container with ID starting with b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a not found: ID does not exist" containerID="b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a"
Oct 30 22:54:31.391489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.391373    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a"} err="failed to get container status \"b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a\": rpc error: code = NotFound desc = could not find container \"b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a\": container with ID starting with b008ccfcf40bafaecfbce8a339f4d11ae17277fa7bb626f3dc7fbb3f104fc74a not found: ID does not exist"
Oct 30 22:54:31.391489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.391398    5503 scope.go:122] "RemoveContainer" containerID="679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf"
Oct 30 22:54:31.401282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.401263    5503 scope.go:122] "RemoveContainer" containerID="679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf"
Oct 30 22:54:31.401679 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:31.401624    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf\": container with ID starting with 679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf not found: ID does not exist" containerID="679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf"
Oct 30 22:54:31.401761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.401691    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf"} err="failed to get container status \"679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf\": rpc error: code = NotFound desc = could not find container \"679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf\": container with ID starting with 679e27c97c7740dc53f65f751a89bd63c82c015add449dd467f3fc187d8440cf not found: ID does not exist"
Oct 30 22:54:31.401761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.401724    5503 scope.go:122] "RemoveContainer" containerID="775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed"
Oct 30 22:54:31.410513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.410489    5503 scope.go:122] "RemoveContainer" containerID="775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed"
Oct 30 22:54:31.410907 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:31.410881    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed\": container with ID starting with 775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed not found: ID does not exist" containerID="775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed"
Oct 30 22:54:31.410989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.410909    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed"} err="failed to get container status \"775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed\": rpc error: code = NotFound desc = could not find container \"775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed\": container with ID starting with 775ddea4ef46aaedc832995d49592ef04342284d9ab49f1e62c8dd4723afcfed not found: ID does not exist"
Oct 30 22:54:31.592420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.592394    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"]
Oct 30 22:54:31.593617 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:54:31.593590    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59b1fa2_41ab_459f_b139_3cfc31c7d521.slice/crio-7a1df7fde0ba5d933e9cc1f30f7a5df2e23c9aa5dd2c2f68ad96fa92672e4f1e WatchSource:0}: Error finding container 7a1df7fde0ba5d933e9cc1f30f7a5df2e23c9aa5dd2c2f68ad96fa92672e4f1e: Status 404 returned error can't find the container with id 7a1df7fde0ba5d933e9cc1f30f7a5df2e23c9aa5dd2c2f68ad96fa92672e4f1e
Oct 30 22:54:31.600544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.600521    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"]
Oct 30 22:54:31.609239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.609221    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/premium-w3-b47e8d-kserve-6c5959568f-hd8jr"]
Oct 30 22:54:31.637510 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.637489    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"]
Oct 30 22:54:31.638445 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:54:31.638422    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999c1eff_d672_4255_92f5_eb3076411333.slice/crio-42cd89fd2315479048733cade74c6ed466a895cedc4621d3c7d1dd4051ea6feb WatchSource:0}: Error finding container 42cd89fd2315479048733cade74c6ed466a895cedc4621d3c7d1dd4051ea6feb: Status 404 returned error can't find the container with id 42cd89fd2315479048733cade74c6ed466a895cedc4621d3c7d1dd4051ea6feb
Oct 30 22:54:31.714061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.714001    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"]
Oct 30 22:54:31.721877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.721761    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf"]
Oct 30 22:54:31.789345 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.789312    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9114be59-0719-48bc-9695-933cf4606b43" path="/var/lib/kubelet/pods/9114be59-0719-48bc-9695-933cf4606b43/volumes"
Oct 30 22:54:31.789762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.789747    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" path="/var/lib/kubelet/pods/e8c62843-bc04-4d6a-a63e-ca97b65f9fc4/volumes"
Oct 30 22:54:31.822585 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.822558    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"]
Oct 30 22:54:31.826745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.826723    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-2-w3-b47e8d-kserve-d7c946555-cdc74"]
Oct 30 22:54:31.926389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.926357    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b47e8d/simulator-w3-b47e8d-kserve-5d64874f84-jbqwf" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:54:31.931095 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.930989    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"]
Oct 30 22:54:31.937261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:31.937237    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b47e8d/distinct-w3-b47e8d-kserve-76878bcf9b-mvwdv"]
Oct 30 22:54:32.342793 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.342703    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8" event={"ID":"999c1eff-d672-4255-92f5-eb3076411333","Type":"ContainerStarted","Data":"09ba91fa8e49f33f7066f40f9624212cbd72044fddf0e5ffcbcb30f6120188e0"}
Oct 30 22:54:32.342943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.342792    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8" event={"ID":"999c1eff-d672-4255-92f5-eb3076411333","Type":"ContainerStarted","Data":"42cd89fd2315479048733cade74c6ed466a895cedc4621d3c7d1dd4051ea6feb"}
Oct 30 22:54:32.344327 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.344299    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs" event={"ID":"e59b1fa2-41ab-459f-b139-3cfc31c7d521","Type":"ContainerStarted","Data":"7e7f70510678b02ca08cbacca994b39154174cdd453c8db4d8ff6f4498186b43"}
Oct 30 22:54:32.344514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.344352    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs" event={"ID":"e59b1fa2-41ab-459f-b139-3cfc31c7d521","Type":"ContainerStarted","Data":"7a1df7fde0ba5d933e9cc1f30f7a5df2e23c9aa5dd2c2f68ad96fa92672e4f1e"}
Oct 30 22:54:32.395418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.395385    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:32.464589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.464535    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8" podStartSLOduration=2.464521983 podStartE2EDuration="2.464521983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:32.462328889 +0000 UTC m=+4048.796905662" watchObservedRunningTime="2026-10-30 22:54:32.464521983 +0000 UTC m=+4048.799098751"
Oct 30 22:54:32.575702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.575651    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs" podStartSLOduration=2.575620422 podStartE2EDuration="2.575620422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:32.574733521 +0000 UTC m=+4048.909310290" watchObservedRunningTime="2026-10-30 22:54:32.575620422 +0000 UTC m=+4048.910197190"
Oct 30 22:54:32.675747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.675711    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:32.827994 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.827963    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:32.906944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.906861    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"]
Oct 30 22:54:32.911649 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:32.911611    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-1b5075-sm8wj"]
Oct 30 22:54:33.355969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.355889    5503 generic.go:356] "Generic (PLEG): container finished" podID="999c1eff-d672-4255-92f5-eb3076411333" containerID="09ba91fa8e49f33f7066f40f9624212cbd72044fddf0e5ffcbcb30f6120188e0" exitCode=0
Oct 30 22:54:33.356141 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.355981    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8" event={"ID":"999c1eff-d672-4255-92f5-eb3076411333","Type":"ContainerDied","Data":"09ba91fa8e49f33f7066f40f9624212cbd72044fddf0e5ffcbcb30f6120188e0"}
Oct 30 22:54:33.357702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.357679    5503 generic.go:356] "Generic (PLEG): container finished" podID="e59b1fa2-41ab-459f-b139-3cfc31c7d521" containerID="7e7f70510678b02ca08cbacca994b39154174cdd453c8db4d8ff6f4498186b43" exitCode=0
Oct 30 22:54:33.357818 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.357768    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs" event={"ID":"e59b1fa2-41ab-459f-b139-3cfc31c7d521","Type":"ContainerDied","Data":"7e7f70510678b02ca08cbacca994b39154174cdd453c8db4d8ff6f4498186b43"}
Oct 30 22:54:33.389381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.389352    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:33.789980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.789948    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14621d87-6e5c-4074-a8ed-b4728e749832" path="/var/lib/kubelet/pods/14621d87-6e5c-4074-a8ed-b4728e749832/volumes"
Oct 30 22:54:33.790407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.790380    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07b7d4d-0016-477f-8edc-47db1175e93b" path="/var/lib/kubelet/pods/d07b7d4d-0016-477f-8edc-47db1175e93b/volumes"
Oct 30 22:54:33.790827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.790812    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" path="/var/lib/kubelet/pods/fc0dec83-8177-4bab-82dc-d62f1d5a7258/volumes"
Oct 30 22:54:33.913868 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:33.913831    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:34.500610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.500589    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:34.537609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.537587    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:34.598511 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.598486    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999c1eff-d672-4255-92f5-eb3076411333-kube-api-access-kg7vr\" (UniqueName: \"kubernetes.io/projected/999c1eff-d672-4255-92f5-eb3076411333-kube-api-access-kg7vr\") pod \"999c1eff-d672-4255-92f5-eb3076411333\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") "
Oct 30 22:54:34.598702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.598526    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-openshift-ca\") pod \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") "
Oct 30 22:54:34.598702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.598574    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-xks-ca\") pod \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") "
Oct 30 22:54:34.598702 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.598619    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c1eff-d672-4255-92f5-eb3076411333-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c1eff-d672-4255-92f5-eb3076411333-maas-api-openshift-ca\") pod \"999c1eff-d672-4255-92f5-eb3076411333\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") "
Oct 30 22:54:34.598894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.598770    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59b1fa2-41ab-459f-b139-3cfc31c7d521-kube-api-access-z5gb4\" (UniqueName: \"kubernetes.io/projected/e59b1fa2-41ab-459f-b139-3cfc31c7d521-kube-api-access-z5gb4\") pod \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\" (UID: \"e59b1fa2-41ab-459f-b139-3cfc31c7d521\") "
Oct 30 22:54:34.598894 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.598805    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c1eff-d672-4255-92f5-eb3076411333-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c1eff-d672-4255-92f5-eb3076411333-maas-api-xks-ca\") pod \"999c1eff-d672-4255-92f5-eb3076411333\" (UID: \"999c1eff-d672-4255-92f5-eb3076411333\") "
Oct 30 22:54:34.599008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.598986    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-openshift-ca" pod "e59b1fa2-41ab-459f-b139-3cfc31c7d521" (UID: "e59b1fa2-41ab-459f-b139-3cfc31c7d521"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:34.599061 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.599034    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c1eff-d672-4255-92f5-eb3076411333-maas-api-openshift-ca" pod "999c1eff-d672-4255-92f5-eb3076411333" (UID: "999c1eff-d672-4255-92f5-eb3076411333"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:34.599201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.599181    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:34.599273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.599206    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c1eff-d672-4255-92f5-eb3076411333-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:34.600823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.600798    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59b1fa2-41ab-459f-b139-3cfc31c7d521-kube-api-access-z5gb4" pod "e59b1fa2-41ab-459f-b139-3cfc31c7d521" (UID: "e59b1fa2-41ab-459f-b139-3cfc31c7d521"). InnerVolumeSpecName "kube-api-access-z5gb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:34.600917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.600807    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c1eff-d672-4255-92f5-eb3076411333-maas-api-xks-ca" pod "999c1eff-d672-4255-92f5-eb3076411333" (UID: "999c1eff-d672-4255-92f5-eb3076411333"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:34.600917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.600832    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999c1eff-d672-4255-92f5-eb3076411333-kube-api-access-kg7vr" pod "999c1eff-d672-4255-92f5-eb3076411333" (UID: "999c1eff-d672-4255-92f5-eb3076411333"). InnerVolumeSpecName "kube-api-access-kg7vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:34.600917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.600831    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-xks-ca" pod "e59b1fa2-41ab-459f-b139-3cfc31c7d521" (UID: "e59b1fa2-41ab-459f-b139-3cfc31c7d521"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:34.700238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.700203    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b1fa2-41ab-459f-b139-3cfc31c7d521-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:34.700238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.700237    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5gb4\" (UniqueName: \"kubernetes.io/projected/e59b1fa2-41ab-459f-b139-3cfc31c7d521-kube-api-access-z5gb4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:34.700461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.700252    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c1eff-d672-4255-92f5-eb3076411333-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:34.700461 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:34.700262    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg7vr\" (UniqueName: \"kubernetes.io/projected/999c1eff-d672-4255-92f5-eb3076411333-kube-api-access-kg7vr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:35.371861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:35.371830    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8"
Oct 30 22:54:35.372279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:35.371829    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e70155-c4lk8" event={"ID":"999c1eff-d672-4255-92f5-eb3076411333","Type":"ContainerDied","Data":"42cd89fd2315479048733cade74c6ed466a895cedc4621d3c7d1dd4051ea6feb"}
Oct 30 22:54:35.372279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:35.371959    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42cd89fd2315479048733cade74c6ed466a895cedc4621d3c7d1dd4051ea6feb"
Oct 30 22:54:35.373324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:35.373303    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs"
Oct 30 22:54:35.373324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:35.373316    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d90c6b12-kgwcs" event={"ID":"e59b1fa2-41ab-459f-b139-3cfc31c7d521","Type":"ContainerDied","Data":"7a1df7fde0ba5d933e9cc1f30f7a5df2e23c9aa5dd2c2f68ad96fa92672e4f1e"}
Oct 30 22:54:35.373541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:35.373361    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a1df7fde0ba5d933e9cc1f30f7a5df2e23c9aa5dd2c2f68ad96fa92672e4f1e"
Oct 30 22:54:35.893158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:35.893120    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:36.866268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:36.866234    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.155212 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:37.155180    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.325132 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:37.325094    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.479420 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:37.479335    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.799734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:37.799625    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:37.969655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:37.969599    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:38.349110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:38.349076    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:38.553712 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:38.553679    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:38.711574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:38.711531    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:38.807498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:38.807465    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"]
Oct 30 22:54:38.853908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:38.853872    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e524ce-2r26k"]
Oct 30 22:54:38.958036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:38.957996    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:39.126098 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:39.126024    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:39.280500 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:39.280468    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:39.485336 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:39.485305    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:39.641974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:39.641936    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:39.788025 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:39.787936    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0204fecc-b36b-46a3-b299-5573da2237bd" path="/var/lib/kubelet/pods/0204fecc-b36b-46a3-b299-5573da2237bd/volumes"
Oct 30 22:54:41.021810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:41.021779    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:41.172532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:41.172471    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:41.390057 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:41.389950    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:42.144552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.144512    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"]
Oct 30 22:54:42.145022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145005    5503 cpu_manager.go:414] "containerMap: removing container" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main"
Oct 30 22:54:42.145063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145025    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main"
Oct 30 22:54:42.145063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145046    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main"
Oct 30 22:54:42.145063 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145051    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145067    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e59b1fa2-41ab-459f-b139-3cfc31c7d521" containerName="revoke-keys"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145074    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59b1fa2-41ab-459f-b139-3cfc31c7d521" containerName="revoke-keys"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145084    5503 cpu_manager.go:414] "containerMap: removing container" podUID="d07b7d4d-0016-477f-8edc-47db1175e93b" containerName="main"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145089    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07b7d4d-0016-477f-8edc-47db1175e93b" containerName="main"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145096    5503 cpu_manager.go:414] "containerMap: removing container" podUID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" containerName="main"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145101    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" containerName="main"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145108    5503 cpu_manager.go:414] "containerMap: removing container" podUID="999c1eff-d672-4255-92f5-eb3076411333" containerName="revoke-keys"
Oct 30 22:54:42.145175 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145112    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c1eff-d672-4255-92f5-eb3076411333" containerName="revoke-keys"
Oct 30 22:54:42.145421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145182    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="9114be59-0719-48bc-9695-933cf4606b43" containerName="main"
Oct 30 22:54:42.145421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145192    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c62843-bc04-4d6a-a63e-ca97b65f9fc4" containerName="main"
Oct 30 22:54:42.145421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145199    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59b1fa2-41ab-459f-b139-3cfc31c7d521" containerName="revoke-keys"
Oct 30 22:54:42.145421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145205    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07b7d4d-0016-477f-8edc-47db1175e93b" containerName="main"
Oct 30 22:54:42.145421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145211    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0dec83-8177-4bab-82dc-d62f1d5a7258" containerName="main"
Oct 30 22:54:42.145421 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.145216    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c1eff-d672-4255-92f5-eb3076411333" containerName="revoke-keys"
Oct 30 22:54:42.148427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.148406    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.186082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.186039    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.186082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.186082    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.186364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.186171    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vssc\" (UniqueName: \"kubernetes.io/projected/fcb1fd79-cac4-44e3-9e1d-c81c79680141-kube-api-access-8vssc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.203908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.203869    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"]
Oct 30 22:54:42.248089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.248050    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:42.248271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.248208    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:42.248271 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.248263    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:42.248358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.248307    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:54:42.287498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.287457    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.287745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.287519    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vssc\" (UniqueName: \"kubernetes.io/projected/fcb1fd79-cac4-44e3-9e1d-c81c79680141-kube-api-access-8vssc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.287745 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.287689    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.288158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.288134    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.289985 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.289962    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.346528 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.346494    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vssc\" (UniqueName: \"kubernetes.io/projected/fcb1fd79-cac4-44e3-9e1d-c81c79680141-kube-api-access-8vssc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.461361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.461326    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:42.833544 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:42.833510    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"]
Oct 30 22:54:42.834997 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:54:42.834973    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb1fd79_cac4_44e3_9e1d_c81c79680141.slice/crio-9135f6ed9b98a5a6c48032f3beceff585bf1799715a7e1a6ac7646e50b5470c5 WatchSource:0}: Error finding container 9135f6ed9b98a5a6c48032f3beceff585bf1799715a7e1a6ac7646e50b5470c5: Status 404 returned error can't find the container with id 9135f6ed9b98a5a6c48032f3beceff585bf1799715a7e1a6ac7646e50b5470c5
Oct 30 22:54:43.424961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:43.424926    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9" event={"ID":"fcb1fd79-cac4-44e3-9e1d-c81c79680141","Type":"ContainerStarted","Data":"97e6d9828c07284d67b1250153680b2392dea96a3909b79c5de4d5aef61589a5"}
Oct 30 22:54:43.425337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:43.424980    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9" event={"ID":"fcb1fd79-cac4-44e3-9e1d-c81c79680141","Type":"ContainerStarted","Data":"9135f6ed9b98a5a6c48032f3beceff585bf1799715a7e1a6ac7646e50b5470c5"}
Oct 30 22:54:43.536896 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:43.536846    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9" podStartSLOduration=1.536829921 podStartE2EDuration="1.536829921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:43.534709957 +0000 UTC m=+4059.869286725" watchObservedRunningTime="2026-10-30 22:54:43.536829921 +0000 UTC m=+4059.871406692"
Oct 30 22:54:43.580715 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:43.580684    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:44.433008 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:44.432971    5503 generic.go:356] "Generic (PLEG): container finished" podID="fcb1fd79-cac4-44e3-9e1d-c81c79680141" containerID="97e6d9828c07284d67b1250153680b2392dea96a3909b79c5de4d5aef61589a5" exitCode=0
Oct 30 22:54:44.433414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:44.433055    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9" event={"ID":"fcb1fd79-cac4-44e3-9e1d-c81c79680141","Type":"ContainerDied","Data":"97e6d9828c07284d67b1250153680b2392dea96a3909b79c5de4d5aef61589a5"}
Oct 30 22:54:45.020130 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.020094    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:45.174338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.174308    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:45.578308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.578280    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:45.620733 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.620704    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb1fd79-cac4-44e3-9e1d-c81c79680141-kube-api-access-8vssc\" (UniqueName: \"kubernetes.io/projected/fcb1fd79-cac4-44e3-9e1d-c81c79680141-kube-api-access-8vssc\") pod \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") "
Oct 30 22:54:45.620869 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.620750    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-openshift-ca\") pod \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") "
Oct 30 22:54:45.620922 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.620891    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-xks-ca\") pod \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\" (UID: \"fcb1fd79-cac4-44e3-9e1d-c81c79680141\") "
Oct 30 22:54:45.621121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.621098    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-openshift-ca" pod "fcb1fd79-cac4-44e3-9e1d-c81c79680141" (UID: "fcb1fd79-cac4-44e3-9e1d-c81c79680141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:45.621242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.621220    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:45.622969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.622942    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb1fd79-cac4-44e3-9e1d-c81c79680141-kube-api-access-8vssc" pod "fcb1fd79-cac4-44e3-9e1d-c81c79680141" (UID: "fcb1fd79-cac4-44e3-9e1d-c81c79680141"). InnerVolumeSpecName "kube-api-access-8vssc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:45.623216 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.622975    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-xks-ca" pod "fcb1fd79-cac4-44e3-9e1d-c81c79680141" (UID: "fcb1fd79-cac4-44e3-9e1d-c81c79680141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:45.704418 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.704347    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:45.721823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.721794    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcb1fd79-cac4-44e3-9e1d-c81c79680141-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:45.721823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.721817    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vssc\" (UniqueName: \"kubernetes.io/projected/fcb1fd79-cac4-44e3-9e1d-c81c79680141-kube-api-access-8vssc\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:45.915600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:45.915555    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:46.075936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:46.075858    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:46.247096 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:46.247055    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:46.446442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:46.446413    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"
Oct 30 22:54:46.446442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:46.446436    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9" event={"ID":"fcb1fd79-cac4-44e3-9e1d-c81c79680141","Type":"ContainerDied","Data":"9135f6ed9b98a5a6c48032f3beceff585bf1799715a7e1a6ac7646e50b5470c5"}
Oct 30 22:54:46.446764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:46.446485    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9135f6ed9b98a5a6c48032f3beceff585bf1799715a7e1a6ac7646e50b5470c5"
Oct 30 22:54:48.629158 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:48.629122    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:52.694889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:52.694849    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:52.861608 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:52.861569    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:53.066405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:53.066314    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:53.262826 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:53.262794    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:53.452031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:53.451996    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"]
Oct 30 22:54:53.580452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:53.580412    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:53.768523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:53.768443    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:54.911214 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:54.911177    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:55.114722 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:55.114692    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:57.931511 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:57.931474    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:58.098736 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:54:58.098689    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:54:58.270996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:58.270912    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:58.388747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:58.388710    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"]
Oct 30 22:54:58.436937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:58.436902    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9717a8-6s2f9"]
Oct 30 22:54:58.452436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:58.452396    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" podUID="69459b23-55f8-4c8b-8638-5f52087ea46f" containerName="payload-pre-processing" containerID="cri-o://e2ee731ca1624ad53ea1501bd4ebac96cb11f0d5c8d31c0843c7e63c8b42d688" gracePeriod=25
Oct 30 22:54:58.871654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:58.871610    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:59.368920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.368881    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:59.580739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.580705    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:59.788276 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.788231    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb1fd79-cac4-44e3-9e1d-c81c79680141" path="/var/lib/kubelet/pods/fcb1fd79-cac4-44e3-9e1d-c81c79680141/volumes"
Oct 30 22:54:59.798452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.798422    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:54:59.854058 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.854030    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"]
Oct 30 22:54:59.854473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.854458    5503 cpu_manager.go:414] "containerMap: removing container" podUID="fcb1fd79-cac4-44e3-9e1d-c81c79680141" containerName="revoke-keys"
Oct 30 22:54:59.854473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.854474    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb1fd79-cac4-44e3-9e1d-c81c79680141" containerName="revoke-keys"
Oct 30 22:54:59.854580 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.854570    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb1fd79-cac4-44e3-9e1d-c81c79680141" containerName="revoke-keys"
Oct 30 22:54:59.859337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.859317    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:54:59.912522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.912488    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"]
Oct 30 22:54:59.957077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.957048    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:59.957077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.957071    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:54:59.957077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.957091    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:54:59.957077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.957114    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:54:59.962972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.962946    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:54:59.963164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.963145    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdws\" (UniqueName: \"kubernetes.io/projected/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-kube-api-access-4xdws\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:54:59.963209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:54:59.963189    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.064375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.064292    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdws\" (UniqueName: \"kubernetes.io/projected/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-kube-api-access-4xdws\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.064375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.064330    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.064740 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.064409    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.064973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.064954    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.066713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.066692    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.122914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.122886    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdws\" (UniqueName: \"kubernetes.io/projected/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-kube-api-access-4xdws\") pod \"maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.173153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.173118    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:00.539530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:00.539502    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"]
Oct 30 22:55:00.541538 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:55:00.541510    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcfc98d_704a_4d88_9bee_35e7bbb91e7f.slice/crio-f310479a292f425fab697b674781d3694baece243029fa3b6e1196fbedda77b5 WatchSource:0}: Error finding container f310479a292f425fab697b674781d3694baece243029fa3b6e1196fbedda77b5: Status 404 returned error can't find the container with id f310479a292f425fab697b674781d3694baece243029fa3b6e1196fbedda77b5
Oct 30 22:55:01.240571 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:01.240539    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:01.396921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:01.396897    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:01.540919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:01.540837    5503 generic.go:356] "Generic (PLEG): container finished" podID="6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" containerID="ea37cd89694b07da536ee5d5a4dba1d6554e95655512067f1c0ffed9ce4e157a" exitCode=0
Oct 30 22:55:01.540919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:01.540881    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx" event={"ID":"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f","Type":"ContainerDied","Data":"ea37cd89694b07da536ee5d5a4dba1d6554e95655512067f1c0ffed9ce4e157a"}
Oct 30 22:55:01.541343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:01.540927    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx" event={"ID":"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f","Type":"ContainerStarted","Data":"f310479a292f425fab697b674781d3694baece243029fa3b6e1196fbedda77b5"}
Oct 30 22:55:02.683817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.683787    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:02.792764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.792732    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-kube-api-access-4xdws\" (UniqueName: \"kubernetes.io/projected/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-kube-api-access-4xdws\") pod \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") "
Oct 30 22:55:02.792965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.792794    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-xks-ca\") pod \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") "
Oct 30 22:55:02.792965 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.792866    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-openshift-ca\") pod \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\" (UID: \"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f\") "
Oct 30 22:55:02.793423 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.793390    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-openshift-ca" pod "6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" (UID: "6fcfc98d-704a-4d88-9bee-35e7bbb91e7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:02.795318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.795292    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-kube-api-access-4xdws" pod "6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" (UID: "6fcfc98d-704a-4d88-9bee-35e7bbb91e7f"). InnerVolumeSpecName "kube-api-access-4xdws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:02.795576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.795554    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-xks-ca" pod "6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" (UID: "6fcfc98d-704a-4d88-9bee-35e7bbb91e7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:02.893859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.893826    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xdws\" (UniqueName: \"kubernetes.io/projected/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-kube-api-access-4xdws\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:02.893859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.893855    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:02.893859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:02.893866    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:03.554997 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:03.554963    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"
Oct 30 22:55:03.555191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:03.554965    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx" event={"ID":"6fcfc98d-704a-4d88-9bee-35e7bbb91e7f","Type":"ContainerDied","Data":"f310479a292f425fab697b674781d3694baece243029fa3b6e1196fbedda77b5"}
Oct 30 22:55:03.555191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:03.555087    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f310479a292f425fab697b674781d3694baece243029fa3b6e1196fbedda77b5"
Oct 30 22:55:04.413321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:04.413280    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:05.226564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:05.226522    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:05.383059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:05.383022    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:05.559909 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:05.559828    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:06.215595 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.215564    5503 scope.go:122] "RemoveContainer" containerID="ed841193b345686ce2be69015c4365c97f191a4d9c1b8e11594eb9f3a15efe92"
Oct 30 22:55:06.224779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.224759    5503 scope.go:122] "RemoveContainer" containerID="8fbddd3a4b98757c8658760dec57b9e2f99c972f1b5f300ab0457d3463840566"
Oct 30 22:55:06.383477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.383443    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"]
Oct 30 22:55:06.383919 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.383906    5503 cpu_manager.go:414] "containerMap: removing container" podUID="6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" containerName="revoke-keys"
Oct 30 22:55:06.383980 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.383921    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" containerName="revoke-keys"
Oct 30 22:55:06.384017 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.383991    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" containerName="revoke-keys"
Oct 30 22:55:06.388910 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.388889    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.486036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.485961    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:06.486036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.486002    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:06.486285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.486269    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:06.486382 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.486367    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:55:06.488698 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.488674    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"]
Oct 30 22:55:06.532106 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.532080    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkpr\" (UniqueName: \"kubernetes.io/projected/8d68fe9f-1800-492d-8ec0-0f7a00b02547-kube-api-access-4wkpr\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.532260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.532144    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.532260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.532217    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.634835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.633659    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.634835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.633718    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.634835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.633838    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkpr\" (UniqueName: \"kubernetes.io/projected/8d68fe9f-1800-492d-8ec0-0f7a00b02547-kube-api-access-4wkpr\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.634835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.634631    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.637082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.637057    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.692817 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.692780    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkpr\" (UniqueName: \"kubernetes.io/projected/8d68fe9f-1800-492d-8ec0-0f7a00b02547-kube-api-access-4wkpr\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.701597 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.701574    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:06.957912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.957878    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" podUID="69459b23-55f8-4c8b-8638-5f52087ea46f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:06.989800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:06.989767    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:55:07.105256 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:07.105226    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"]
Oct 30 22:55:07.107252 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:55:07.107211    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d68fe9f_1800_492d_8ec0_0f7a00b02547.slice/crio-2c5f88f10c682b58f660a85fca393bc0e464196e453cc9f53c9140b0c907037d WatchSource:0}: Error finding container 2c5f88f10c682b58f660a85fca393bc0e464196e453cc9f53c9140b0c907037d: Status 404 returned error can't find the container with id 2c5f88f10c682b58f660a85fca393bc0e464196e453cc9f53c9140b0c907037d
Oct 30 22:55:07.586011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:07.585966    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws" event={"ID":"8d68fe9f-1800-492d-8ec0-0f7a00b02547","Type":"ContainerStarted","Data":"963b3957d5553ee3c1c0efb4da7a26e3334af02e6beae2c6cd1f36be3ec92589"}
Oct 30 22:55:07.586231 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:07.586033    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws" event={"ID":"8d68fe9f-1800-492d-8ec0-0f7a00b02547","Type":"ContainerStarted","Data":"2c5f88f10c682b58f660a85fca393bc0e464196e453cc9f53c9140b0c907037d"}
Oct 30 22:55:07.700221 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:07.700118    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws" podStartSLOduration=1.700102095 podStartE2EDuration="1.700102095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:07.699083547 +0000 UTC m=+4084.033660315" watchObservedRunningTime="2026-10-30 22:55:07.700102095 +0000 UTC m=+4084.034678925"
Oct 30 22:55:08.593545 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:08.593505    5503 generic.go:356] "Generic (PLEG): container finished" podID="8d68fe9f-1800-492d-8ec0-0f7a00b02547" containerID="963b3957d5553ee3c1c0efb4da7a26e3334af02e6beae2c6cd1f36be3ec92589" exitCode=0
Oct 30 22:55:08.593736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:08.593544    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws" event={"ID":"8d68fe9f-1800-492d-8ec0-0f7a00b02547","Type":"ContainerDied","Data":"963b3957d5553ee3c1c0efb4da7a26e3334af02e6beae2c6cd1f36be3ec92589"}
Oct 30 22:55:09.215618 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.215586    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:09.740192 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.740157    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:09.868937 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.868864    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d68fe9f-1800-492d-8ec0-0f7a00b02547-kube-api-access-4wkpr\" (UniqueName: \"kubernetes.io/projected/8d68fe9f-1800-492d-8ec0-0f7a00b02547-kube-api-access-4wkpr\") pod \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") "
Oct 30 22:55:09.869082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.868987    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-openshift-ca\") pod \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") "
Oct 30 22:55:09.869082 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.869061    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-xks-ca\") pod \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\" (UID: \"8d68fe9f-1800-492d-8ec0-0f7a00b02547\") "
Oct 30 22:55:09.869373 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.869350    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-openshift-ca" pod "8d68fe9f-1800-492d-8ec0-0f7a00b02547" (UID: "8d68fe9f-1800-492d-8ec0-0f7a00b02547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:09.871189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.871151    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d68fe9f-1800-492d-8ec0-0f7a00b02547-kube-api-access-4wkpr" pod "8d68fe9f-1800-492d-8ec0-0f7a00b02547" (UID: "8d68fe9f-1800-492d-8ec0-0f7a00b02547"). InnerVolumeSpecName "kube-api-access-4wkpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:09.871189 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.871179    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-xks-ca" pod "8d68fe9f-1800-492d-8ec0-0f7a00b02547" (UID: "8d68fe9f-1800-492d-8ec0-0f7a00b02547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:09.970023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.969992    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:09.970023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.970019    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wkpr\" (UniqueName: \"kubernetes.io/projected/8d68fe9f-1800-492d-8ec0-0f7a00b02547-kube-api-access-4wkpr\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:09.970023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:09.970031    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d68fe9f-1800-492d-8ec0-0f7a00b02547-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:10.607519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:10.607488    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"
Oct 30 22:55:10.607519 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:10.607512    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws" event={"ID":"8d68fe9f-1800-492d-8ec0-0f7a00b02547","Type":"ContainerDied","Data":"2c5f88f10c682b58f660a85fca393bc0e464196e453cc9f53c9140b0c907037d"}
Oct 30 22:55:10.608097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:10.607563    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5f88f10c682b58f660a85fca393bc0e464196e453cc9f53c9140b0c907037d"
Oct 30 22:55:11.402319 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:11.402285    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"]
Oct 30 22:55:11.513005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:11.512967    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"]
Oct 30 22:55:11.797561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:11.797480    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:11.950386 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:11.950348    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:13.185503 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:13.185472    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:14.722303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:14.722271    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:14.899160 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:14.899128    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:15.081736 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:15.081631    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"]
Oct 30 22:55:15.127700 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:15.127672    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b47e8d-cdkcq"]
Oct 30 22:55:15.524331 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:15.524301    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:15.787912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:15.787839    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135d5792-a394-4f6a-8fdf-7596b4fef503" path="/var/lib/kubelet/pods/135d5792-a394-4f6a-8fdf-7596b4fef503/volumes"
Oct 30 22:55:16.402606 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.402568    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb" podUID="06d20a62-5c5e-4de8-b950-67819409cbc7" containerName="payload-processing" containerID="cri-o://64c18d1dd969388dd6caada9a43ba6a49a08976e9d92e1415a70f3689a0e5bf3" gracePeriod=25
Oct 30 22:55:16.513685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.513620    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2" podUID="3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" containerName="payload-pre-processing" containerID="cri-o://cb8cd522697bb61196d180dafe90871c840e3749f652d36f047ca04cd5714c85" gracePeriod=25
Oct 30 22:55:16.646600 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.646570    5503 generic.go:356] "Generic (PLEG): container finished" podID="06d20a62-5c5e-4de8-b950-67819409cbc7" containerID="64c18d1dd969388dd6caada9a43ba6a49a08976e9d92e1415a70f3689a0e5bf3" exitCode=0
Oct 30 22:55:16.646769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.646747    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb" event={"ID":"06d20a62-5c5e-4de8-b950-67819409cbc7","Type":"ContainerDied","Data":"64c18d1dd969388dd6caada9a43ba6a49a08976e9d92e1415a70f3689a0e5bf3"}
Oct 30 22:55:16.646842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.646807    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb" event={"ID":"06d20a62-5c5e-4de8-b950-67819409cbc7","Type":"ContainerDied","Data":"96b45cfe6a3decbeec61c7ecf8bdb269549acbc9b307aafdb72c38bd9503097b"}
Oct 30 22:55:16.646842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.646833    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96b45cfe6a3decbeec61c7ecf8bdb269549acbc9b307aafdb72c38bd9503097b"
Oct 30 22:55:16.648866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.648844    5503 generic.go:356] "Generic (PLEG): container finished" podID="3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" containerID="cb8cd522697bb61196d180dafe90871c840e3749f652d36f047ca04cd5714c85" exitCode=0
Oct 30 22:55:16.648987 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.648898    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2" event={"ID":"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981","Type":"ContainerDied","Data":"cb8cd522697bb61196d180dafe90871c840e3749f652d36f047ca04cd5714c85"}
Oct 30 22:55:16.660769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.660710    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:55:16.766249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.766226    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:55:16.841951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.841912    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d20a62-5c5e-4de8-b950-67819409cbc7-kube-api-access-6hdwq\" (UniqueName: \"kubernetes.io/projected/06d20a62-5c5e-4de8-b950-67819409cbc7-kube-api-access-6hdwq\") pod \"06d20a62-5c5e-4de8-b950-67819409cbc7\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") "
Oct 30 22:55:16.842411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.842040    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d20a62-5c5e-4de8-b950-67819409cbc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d20a62-5c5e-4de8-b950-67819409cbc7-plugins-config-volume\") pod \"06d20a62-5c5e-4de8-b950-67819409cbc7\" (UID: \"06d20a62-5c5e-4de8-b950-67819409cbc7\") "
Oct 30 22:55:16.842411 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.842380    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d20a62-5c5e-4de8-b950-67819409cbc7-plugins-config-volume" pod "06d20a62-5c5e-4de8-b950-67819409cbc7" (UID: "06d20a62-5c5e-4de8-b950-67819409cbc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:16.844022 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.844001    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d20a62-5c5e-4de8-b950-67819409cbc7-kube-api-access-6hdwq" pod "06d20a62-5c5e-4de8-b950-67819409cbc7" (UID: "06d20a62-5c5e-4de8-b950-67819409cbc7"). InnerVolumeSpecName "kube-api-access-6hdwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:16.943655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.943550    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-plugins-config-volume\") pod \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") "
Oct 30 22:55:16.943655 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.943612    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-kube-api-access-hq9ml\" (UniqueName: \"kubernetes.io/projected/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-kube-api-access-hq9ml\") pod \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\" (UID: \"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981\") "
Oct 30 22:55:16.944019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.943993    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-plugins-config-volume" pod "3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" (UID: "3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:16.944143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.944120    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:16.944143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.944140    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d20a62-5c5e-4de8-b950-67819409cbc7-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:16.944266 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.944150    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hdwq\" (UniqueName: \"kubernetes.io/projected/06d20a62-5c5e-4de8-b950-67819409cbc7-kube-api-access-6hdwq\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:16.945846 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.945828    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-kube-api-access-hq9ml" pod "3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" (UID: "3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981"). InnerVolumeSpecName "kube-api-access-hq9ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:16.961663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:16.958679    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" podUID="69459b23-55f8-4c8b-8638-5f52087ea46f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:17.045150 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.045108    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq9ml\" (UniqueName: \"kubernetes.io/projected/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981-kube-api-access-hq9ml\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:17.658127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.658090    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2" event={"ID":"3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981","Type":"ContainerDied","Data":"6ff6794366acc46108a2790de97b5ccb2d10ba0a6228eff5ea6d8d16e8b229b8"}
Oct 30 22:55:17.658127 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.658114    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"
Oct 30 22:55:17.658514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.658195    5503 scope.go:122] "RemoveContainer" containerID="cb8cd522697bb61196d180dafe90871c840e3749f652d36f047ca04cd5714c85"
Oct 30 22:55:17.658514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.658408    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"
Oct 30 22:55:17.817882 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.817842    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"]
Oct 30 22:55:17.864012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.863980    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9626a6-c8865b65f-gkpv2"]
Oct 30 22:55:17.969430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.969344    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"]
Oct 30 22:55:17.976860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.976833    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9626a6-8547bc8687-5xsjb"]
Oct 30 22:55:17.979603 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:17.979579    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"]
Oct 30 22:55:18.097822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.097789    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"]
Oct 30 22:55:18.367393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.367318    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:18.517213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.517186    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:18.671534 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.671506    5503 generic.go:356] "Generic (PLEG): container finished" podID="69459b23-55f8-4c8b-8638-5f52087ea46f" containerID="e2ee731ca1624ad53ea1501bd4ebac96cb11f0d5c8d31c0843c7e63c8b42d688" exitCode=0
Oct 30 22:55:18.671744 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.671577    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" event={"ID":"69459b23-55f8-4c8b-8638-5f52087ea46f","Type":"ContainerDied","Data":"e2ee731ca1624ad53ea1501bd4ebac96cb11f0d5c8d31c0843c7e63c8b42d688"}
Oct 30 22:55:18.712505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.712476    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:55:18.867012 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.866976    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69459b23-55f8-4c8b-8638-5f52087ea46f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69459b23-55f8-4c8b-8638-5f52087ea46f-plugins-config-volume\") pod \"69459b23-55f8-4c8b-8638-5f52087ea46f\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") "
Oct 30 22:55:18.867470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.867071    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69459b23-55f8-4c8b-8638-5f52087ea46f-kube-api-access-6c5gh\" (UniqueName: \"kubernetes.io/projected/69459b23-55f8-4c8b-8638-5f52087ea46f-kube-api-access-6c5gh\") pod \"69459b23-55f8-4c8b-8638-5f52087ea46f\" (UID: \"69459b23-55f8-4c8b-8638-5f52087ea46f\") "
Oct 30 22:55:18.867470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.867324    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69459b23-55f8-4c8b-8638-5f52087ea46f-plugins-config-volume" pod "69459b23-55f8-4c8b-8638-5f52087ea46f" (UID: "69459b23-55f8-4c8b-8638-5f52087ea46f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:18.867470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.867435    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69459b23-55f8-4c8b-8638-5f52087ea46f-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:18.869128 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.869101    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69459b23-55f8-4c8b-8638-5f52087ea46f-kube-api-access-6c5gh" pod "69459b23-55f8-4c8b-8638-5f52087ea46f" (UID: "69459b23-55f8-4c8b-8638-5f52087ea46f"). InnerVolumeSpecName "kube-api-access-6c5gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:18.968428 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:18.968387    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c5gh\" (UniqueName: \"kubernetes.io/projected/69459b23-55f8-4c8b-8638-5f52087ea46f-kube-api-access-6c5gh\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:19.526023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.525984    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:19.682644 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.682604    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"
Oct 30 22:55:19.682845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.682628    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2" event={"ID":"69459b23-55f8-4c8b-8638-5f52087ea46f","Type":"ContainerDied","Data":"f9e3444dcac4a108f7744aaa3dad792595273762a11dd658ed5bb8156d28df45"}
Oct 30 22:55:19.682845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.682716    5503 scope.go:122] "RemoveContainer" containerID="e2ee731ca1624ad53ea1501bd4ebac96cb11f0d5c8d31c0843c7e63c8b42d688"
Oct 30 22:55:19.788647 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.788563    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d20a62-5c5e-4de8-b950-67819409cbc7" path="/var/lib/kubelet/pods/06d20a62-5c5e-4de8-b950-67819409cbc7/volumes"
Oct 30 22:55:19.788981 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.788967    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" path="/var/lib/kubelet/pods/3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981/volumes"
Oct 30 22:55:19.841587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.841556    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"]
Oct 30 22:55:19.889041 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:19.889011    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9717a8-5b68568959-cs2q2"]
Oct 30 22:55:21.788679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:21.788630    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69459b23-55f8-4c8b-8638-5f52087ea46f" path="/var/lib/kubelet/pods/69459b23-55f8-4c8b-8638-5f52087ea46f/volumes"
Oct 30 22:55:22.019815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:22.019770    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:22.300611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:22.300543    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:22.375054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:22.375018    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"]
Oct 30 22:55:22.375303 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:22.375267    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy" containerID="cri-o://44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5" gracePeriod=30
Oct 30 22:55:22.458879 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:22.458840    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:22.980389 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:22.980348    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d" podUID="19a33ca6-1433-4f69-9617-b9e8c6ca65bf" containerName="payload-processing" containerID="cri-o://143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f" gracePeriod=25
Oct 30 22:55:23.099020 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.098987    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" podUID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" containerName="payload-pre-processing" containerID="cri-o://9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3" gracePeriod=25
Oct 30 22:55:23.231521 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.231460    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:55:23.310139 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.310100    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-kube-api-access-4f4cf\" (UniqueName: \"kubernetes.io/projected/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-kube-api-access-4f4cf\") pod \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") "
Oct 30 22:55:23.310332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.310288    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-plugins-config-volume\") pod \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\" (UID: \"19a33ca6-1433-4f69-9617-b9e8c6ca65bf\") "
Oct 30 22:55:23.310685 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.310658    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-plugins-config-volume" pod "19a33ca6-1433-4f69-9617-b9e8c6ca65bf" (UID: "19a33ca6-1433-4f69-9617-b9e8c6ca65bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:23.312212 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.312196    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-kube-api-access-4f4cf" pod "19a33ca6-1433-4f69-9617-b9e8c6ca65bf" (UID: "19a33ca6-1433-4f69-9617-b9e8c6ca65bf"). InnerVolumeSpecName "kube-api-access-4f4cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:23.411862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.411819    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4cf\" (UniqueName: \"kubernetes.io/projected/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-kube-api-access-4f4cf\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:23.411862 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.411855    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a33ca6-1433-4f69-9617-b9e8c6ca65bf-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:23.708213 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.708177    5503 generic.go:356] "Generic (PLEG): container finished" podID="19a33ca6-1433-4f69-9617-b9e8c6ca65bf" containerID="143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f" exitCode=0
Oct 30 22:55:23.708410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.708224    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"
Oct 30 22:55:23.708410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.708228    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d" event={"ID":"19a33ca6-1433-4f69-9617-b9e8c6ca65bf","Type":"ContainerDied","Data":"143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f"}
Oct 30 22:55:23.708410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.708332    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d" event={"ID":"19a33ca6-1433-4f69-9617-b9e8c6ca65bf","Type":"ContainerDied","Data":"14628a3ce2811ee4ce1c58998083613cb0f783668652bbfa03d176d6981e8802"}
Oct 30 22:55:23.708410 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.708363    5503 scope.go:122] "RemoveContainer" containerID="143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f"
Oct 30 22:55:23.718741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.718721    5503 scope.go:122] "RemoveContainer" containerID="143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f"
Oct 30 22:55:23.718983 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:55:23.718965    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f\": container with ID starting with 143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f not found: ID does not exist" containerID="143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f"
Oct 30 22:55:23.719034 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.718994    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f"} err="failed to get container status \"143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f\": rpc error: code = NotFound desc = could not find container \"143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f\": container with ID starting with 143864b435d6e28bb8a430dcceb3cdf6145ccb0402dbf0a5bf3d09fdb7a80e2f not found: ID does not exist"
Oct 30 22:55:23.863871 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.863840    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"]
Oct 30 22:55:23.869046 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:23.869015    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-68f80c-85b9b7d5c-9tt9d"]
Oct 30 22:55:24.121755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:24.121676    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:24.142644 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:24.142602    5503 patch_prober.go:29] interesting pod/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:55:24.142804 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:24.142684    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:55:25.788273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:25.788240    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a33ca6-1433-4f69-9617-b9e8c6ca65bf" path="/var/lib/kubelet/pods/19a33ca6-1433-4f69-9617-b9e8c6ca65bf/volumes"
Oct 30 22:55:27.623720 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.623697    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:55:27.653249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653221    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-credential-socket\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653255    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-envoy\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653333    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-data\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653352    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istiod-ca-cert\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653413 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653387    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-token\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-token\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653418    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-socket\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653470    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-kube-api-access-264zf\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-kube-api-access-264zf\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653505    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-podinfo\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653511    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-credential-socket" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:27.653679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653549    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-certs\") pod \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\" (UID: \"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1\") "
Oct 30 22:55:27.653679 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653616    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-data" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:27.654088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.653777    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istiod-ca-cert" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:27.654088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.654001    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-certs" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:27.654202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.654095    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-socket" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:27.654202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.654121    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.654202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.654142    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.654202 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.654154    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.655797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.655768    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-podinfo" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:55:27.655797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.655778    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-token" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:27.655930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.655884    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-envoy" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:55:27.655930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.655904    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-kube-api-access-264zf" pod "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" (UID: "dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1"). InnerVolumeSpecName "kube-api-access-264zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:27.735768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.735732    5503 generic.go:356] "Generic (PLEG): container finished" podID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerID="44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5" exitCode=0
Oct 30 22:55:27.735972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.735782    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" event={"ID":"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1","Type":"ContainerDied","Data":"44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5"}
Oct 30 22:55:27.735972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.735800    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"
Oct 30 22:55:27.735972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.735861    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln" event={"ID":"dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1","Type":"ContainerDied","Data":"25eb99d6f64ed97e34f41284cd1c1e2f8c4fa915a9d94a162916b86119d05683"}
Oct 30 22:55:27.735972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.735885    5503 scope.go:122] "RemoveContainer" containerID="44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5"
Oct 30 22:55:27.746530 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.746502    5503 scope.go:122] "RemoveContainer" containerID="44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5"
Oct 30 22:55:27.746817 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:55:27.746796    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5\": container with ID starting with 44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5 not found: ID does not exist" containerID="44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5"
Oct 30 22:55:27.746902 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.746830    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5"} err="failed to get container status \"44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5\": rpc error: code = NotFound desc = could not find container \"44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5\": container with ID starting with 44ce5528fd400b422c83a2df19042ec8aabb1a44824f59245235f90c1a9d5db5 not found: ID does not exist"
Oct 30 22:55:27.755326 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.755306    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264zf\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-kube-api-access-264zf\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.755417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.755328    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.755417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.755338    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.755417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.755348    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.755417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.755356    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.755417 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.755365    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:27.899340 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.899293    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"]
Oct 30 22:55:27.903769 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:27.903699    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b47e8d-openshift-default-76b78dfdff-wvrln"]
Oct 30 22:55:28.098125 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:55:28.098093    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:55:28.714911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:28.714877    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:29.788036 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:29.787999    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" path="/var/lib/kubelet/pods/dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1/volumes"
Oct 30 22:55:30.306100 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:30.306067    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" podUID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:32.677290 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:32.677254    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:33.025073 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:33.024983    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:33.740249 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:33.740218    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:55:33.751797 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:33.751766    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:34.096149 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:34.096066    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:34.250914 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:34.250871    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:34.403233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:34.403187    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:34.567300 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:34.567261    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"]
Oct 30 22:55:34.614324 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:34.614292    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9626a6-67tqx"]
Oct 30 22:55:35.032741 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:35.032707    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:35.222653 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:35.222606    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:35.540908 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:35.540871    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:35.705441 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:35.705410    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:35.789195 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:35.789158    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcfc98d-704a-4d88-9bee-35e7bbb91e7f" path="/var/lib/kubelet/pods/6fcfc98d-704a-4d88-9bee-35e7bbb91e7f/volumes"
Oct 30 22:55:35.789477 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:35.789464    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"]
Oct 30 22:55:35.832325 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:35.832246    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-68f80c-cjqws"]
Oct 30 22:55:36.228448 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:36.228405    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:37.788164 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:37.788130    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d68fe9f-1800-492d-8ec0-0f7a00b02547" path="/var/lib/kubelet/pods/8d68fe9f-1800-492d-8ec0-0f7a00b02547/volumes"
Oct 30 22:55:40.305892 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:40.305856    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" podUID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:40.421337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:40.421286    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:41.444377 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:41.444338    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:41.607657 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:41.607595    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:41.814883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:41.814781    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:42.678889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.678852    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"]
Oct 30 22:55:42.679330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679281    5503 cpu_manager.go:414] "containerMap: removing container" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy"
Oct 30 22:55:42.679330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679294    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy"
Oct 30 22:55:42.679330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679303    5503 cpu_manager.go:414] "containerMap: removing container" podUID="69459b23-55f8-4c8b-8638-5f52087ea46f" containerName="payload-pre-processing"
Oct 30 22:55:42.679330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679309    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69459b23-55f8-4c8b-8638-5f52087ea46f" containerName="payload-pre-processing"
Oct 30 22:55:42.679330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679319    5503 cpu_manager.go:414] "containerMap: removing container" podUID="3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" containerName="payload-pre-processing"
Oct 30 22:55:42.679330 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679325    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" containerName="payload-pre-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679338    5503 cpu_manager.go:414] "containerMap: removing container" podUID="8d68fe9f-1800-492d-8ec0-0f7a00b02547" containerName="revoke-keys"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679343    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d68fe9f-1800-492d-8ec0-0f7a00b02547" containerName="revoke-keys"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679357    5503 cpu_manager.go:414] "containerMap: removing container" podUID="06d20a62-5c5e-4de8-b950-67819409cbc7" containerName="payload-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679363    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d20a62-5c5e-4de8-b950-67819409cbc7" containerName="payload-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679373    5503 cpu_manager.go:414] "containerMap: removing container" podUID="19a33ca6-1433-4f69-9617-b9e8c6ca65bf" containerName="payload-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679377    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a33ca6-1433-4f69-9617-b9e8c6ca65bf" containerName="payload-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679443    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafd3c91-fc3b-4ab3-8e1e-6f6ee23a2aa1" containerName="istio-proxy"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679452    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="69459b23-55f8-4c8b-8638-5f52087ea46f" containerName="payload-pre-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679459    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed8f02a-3a4e-4ee9-9c60-d7b7445a8981" containerName="payload-pre-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679466    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d68fe9f-1800-492d-8ec0-0f7a00b02547" containerName="revoke-keys"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679473    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d20a62-5c5e-4de8-b950-67819409cbc7" containerName="payload-processing"
Oct 30 22:55:42.679554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.679480    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a33ca6-1433-4f69-9617-b9e8c6ca65bf" containerName="payload-processing"
Oct 30 22:55:42.682540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.682523    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.737226 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.737197    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"]
Oct 30 22:55:42.782387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.782355    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:42.782387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.782376    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:42.782605 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.782459    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:42.785355 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.785333    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:55:42.810513 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.810480    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.810728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.810535    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cs7r\" (UniqueName: \"kubernetes.io/projected/871c8322-59be-4114-87b2-08d17b5d7acc-kube-api-access-6cs7r\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.810728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.810630    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.911962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.911925    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.912159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.911974    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cs7r\" (UniqueName: \"kubernetes.io/projected/871c8322-59be-4114-87b2-08d17b5d7acc-kube-api-access-6cs7r\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.912159 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.912003    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.912711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.912692    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.914436 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.914413    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.970601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.970526    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cs7r\" (UniqueName: \"kubernetes.io/projected/871c8322-59be-4114-87b2-08d17b5d7acc-kube-api-access-6cs7r\") pod \"maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:42.995497 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:42.995466    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:43.084925 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.084863    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:43.243623 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.243482    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:43.339318 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.339292    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:55:43.363819 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.363791    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"]
Oct 30 22:55:43.365402 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:55:43.365369    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871c8322_59be_4114_87b2_08d17b5d7acc.slice/crio-c815c848bbe7cfe5308929c9c722846724a314f8f0af7e109c9021249b66dc03 WatchSource:0}: Error finding container c815c848bbe7cfe5308929c9c722846724a314f8f0af7e109c9021249b66dc03: Status 404 returned error can't find the container with id c815c848bbe7cfe5308929c9c722846724a314f8f0af7e109c9021249b66dc03
Oct 30 22:55:43.417761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.417735    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8bafda-7d41-4661-800f-d2daf4f40a8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8bafda-7d41-4661-800f-d2daf4f40a8b-plugins-config-volume\") pod \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") "
Oct 30 22:55:43.417926 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.417906    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8bafda-7d41-4661-800f-d2daf4f40a8b-kube-api-access-ffxxk\" (UniqueName: \"kubernetes.io/projected/fb8bafda-7d41-4661-800f-d2daf4f40a8b-kube-api-access-ffxxk\") pod \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\" (UID: \"fb8bafda-7d41-4661-800f-d2daf4f40a8b\") "
Oct 30 22:55:43.418148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.418125    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8bafda-7d41-4661-800f-d2daf4f40a8b-plugins-config-volume" pod "fb8bafda-7d41-4661-800f-d2daf4f40a8b" (UID: "fb8bafda-7d41-4661-800f-d2daf4f40a8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:43.420051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.420029    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8bafda-7d41-4661-800f-d2daf4f40a8b-kube-api-access-ffxxk" pod "fb8bafda-7d41-4661-800f-d2daf4f40a8b" (UID: "fb8bafda-7d41-4661-800f-d2daf4f40a8b"). InnerVolumeSpecName "kube-api-access-ffxxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:43.456002 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.455976    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:43.518779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.518695    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb8bafda-7d41-4661-800f-d2daf4f40a8b-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:43.518779 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.518728    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffxxk\" (UniqueName: \"kubernetes.io/projected/fb8bafda-7d41-4661-800f-d2daf4f40a8b-kube-api-access-ffxxk\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:43.833747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.833710    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn" event={"ID":"871c8322-59be-4114-87b2-08d17b5d7acc","Type":"ContainerStarted","Data":"5a01e638e3602be29f8f5a61fb317295622717495ac05aedf216bd219da6f8dd"}
Oct 30 22:55:43.834215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.833770    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn" event={"ID":"871c8322-59be-4114-87b2-08d17b5d7acc","Type":"ContainerStarted","Data":"c815c848bbe7cfe5308929c9c722846724a314f8f0af7e109c9021249b66dc03"}
Oct 30 22:55:43.835031 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.835006    5503 generic.go:356] "Generic (PLEG): container finished" podID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" containerID="9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3" exitCode=0
Oct 30 22:55:43.835121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.835046    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"
Oct 30 22:55:43.835121 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.835083    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" event={"ID":"fb8bafda-7d41-4661-800f-d2daf4f40a8b","Type":"ContainerDied","Data":"9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3"}
Oct 30 22:55:43.835201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.835132    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6" event={"ID":"fb8bafda-7d41-4661-800f-d2daf4f40a8b","Type":"ContainerDied","Data":"a81e17bddda6eba3f84aa3d56f942ea7cfa4567283f1ad15d8fe8628962124b9"}
Oct 30 22:55:43.835201 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.835156    5503 scope.go:122] "RemoveContainer" containerID="9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3"
Oct 30 22:55:43.844322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.844296    5503 scope.go:122] "RemoveContainer" containerID="9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3"
Oct 30 22:55:43.844644 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:55:43.844611    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3\": container with ID starting with 9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3 not found: ID does not exist" containerID="9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3"
Oct 30 22:55:43.844724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.844651    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3"} err="failed to get container status \"9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3\": rpc error: code = NotFound desc = could not find container \"9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3\": container with ID starting with 9c1db762c28eadae82a0fed33acbf6572f4c3439f9703175cabe667207a261b3 not found: ID does not exist"
Oct 30 22:55:43.970370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:43.970304    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn" podStartSLOduration=1.9702915189999999 podStartE2EDuration="1.970291519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:43.968930292 +0000 UTC m=+4120.303507063" watchObservedRunningTime="2026-10-30 22:55:43.970291519 +0000 UTC m=+4120.304868287"
Oct 30 22:55:44.004619 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.004584    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:44.076604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.076514    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"]
Oct 30 22:55:44.083669 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.083622    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-68f80c-5b57d799c9-rzcr6"]
Oct 30 22:55:44.399672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.399567    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"]
Oct 30 22:55:44.400092 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.400072    5503 cpu_manager.go:414] "containerMap: removing container" podUID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" containerName="payload-pre-processing"
Oct 30 22:55:44.400180 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.400093    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" containerName="payload-pre-processing"
Oct 30 22:55:44.400254 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.400237    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" containerName="payload-pre-processing"
Oct 30 22:55:44.403328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.403305    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.458972 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.458942    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"]
Oct 30 22:55:44.528529 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.528496    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjxj\" (UniqueName: \"kubernetes.io/projected/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-kube-api-access-4bjxj\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.528749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.528625    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.528749 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.528688    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.629755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.629726    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.629918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.629764    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.629918 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.629804    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjxj\" (UniqueName: \"kubernetes.io/projected/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-kube-api-access-4bjxj\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.630484 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.630463    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.631979 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.631961    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.688767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.688742    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjxj\" (UniqueName: \"kubernetes.io/projected/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-kube-api-access-4bjxj\") pod \"maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.715514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.715491    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:44.842897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.842862    5503 generic.go:356] "Generic (PLEG): container finished" podID="871c8322-59be-4114-87b2-08d17b5d7acc" containerID="5a01e638e3602be29f8f5a61fb317295622717495ac05aedf216bd219da6f8dd" exitCode=0
Oct 30 22:55:44.843361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:44.842898    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn" event={"ID":"871c8322-59be-4114-87b2-08d17b5d7acc","Type":"ContainerDied","Data":"5a01e638e3602be29f8f5a61fb317295622717495ac05aedf216bd219da6f8dd"}
Oct 30 22:55:45.098334 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:45.098310    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"]
Oct 30 22:55:45.100126 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:55:45.100083    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8f88a7_86d4_42eb_839b_07b8a29b89b8.slice/crio-b88d2071f34619a176054ebf8ae7f3645acbd0f2df56704ad160e8559ca5cc0c WatchSource:0}: Error finding container b88d2071f34619a176054ebf8ae7f3645acbd0f2df56704ad160e8559ca5cc0c: Status 404 returned error can't find the container with id b88d2071f34619a176054ebf8ae7f3645acbd0f2df56704ad160e8559ca5cc0c
Oct 30 22:55:45.275666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:45.275619    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:45.789610 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:45.789571    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8bafda-7d41-4661-800f-d2daf4f40a8b" path="/var/lib/kubelet/pods/fb8bafda-7d41-4661-800f-d2daf4f40a8b/volumes"
Oct 30 22:55:45.850705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:45.850669    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v" event={"ID":"1c8f88a7-86d4-42eb-839b-07b8a29b89b8","Type":"ContainerStarted","Data":"e7edf1bc2baff758f75c9dfde54f53dd64e1ef5467a718ef5434592989042bc0"}
Oct 30 22:55:45.851110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:45.850723    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v" event={"ID":"1c8f88a7-86d4-42eb-839b-07b8a29b89b8","Type":"ContainerStarted","Data":"b88d2071f34619a176054ebf8ae7f3645acbd0f2df56704ad160e8559ca5cc0c"}
Oct 30 22:55:45.962112 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:45.962064    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v" podStartSLOduration=1.962049769 podStartE2EDuration="1.962049769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:45.960716215 +0000 UTC m=+4122.295292985" watchObservedRunningTime="2026-10-30 22:55:45.962049769 +0000 UTC m=+4122.296626559"
Oct 30 22:55:45.996422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:45.996397    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:46.043508 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.043425    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-xks-ca\") pod \"871c8322-59be-4114-87b2-08d17b5d7acc\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") "
Oct 30 22:55:46.043719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.043552    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871c8322-59be-4114-87b2-08d17b5d7acc-kube-api-access-6cs7r\" (UniqueName: \"kubernetes.io/projected/871c8322-59be-4114-87b2-08d17b5d7acc-kube-api-access-6cs7r\") pod \"871c8322-59be-4114-87b2-08d17b5d7acc\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") "
Oct 30 22:55:46.043719 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.043648    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-openshift-ca\") pod \"871c8322-59be-4114-87b2-08d17b5d7acc\" (UID: \"871c8322-59be-4114-87b2-08d17b5d7acc\") "
Oct 30 22:55:46.043959 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.043936    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-openshift-ca" pod "871c8322-59be-4114-87b2-08d17b5d7acc" (UID: "871c8322-59be-4114-87b2-08d17b5d7acc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:46.045452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.045432    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-xks-ca" pod "871c8322-59be-4114-87b2-08d17b5d7acc" (UID: "871c8322-59be-4114-87b2-08d17b5d7acc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:46.045622 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.045602    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871c8322-59be-4114-87b2-08d17b5d7acc-kube-api-access-6cs7r" pod "871c8322-59be-4114-87b2-08d17b5d7acc" (UID: "871c8322-59be-4114-87b2-08d17b5d7acc"). InnerVolumeSpecName "kube-api-access-6cs7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:46.144934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.144901    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:46.144934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.144931    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cs7r\" (UniqueName: \"kubernetes.io/projected/871c8322-59be-4114-87b2-08d17b5d7acc-kube-api-access-6cs7r\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:46.145148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.144946    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/871c8322-59be-4114-87b2-08d17b5d7acc-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:46.858439 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.858402    5503 generic.go:356] "Generic (PLEG): container finished" podID="1c8f88a7-86d4-42eb-839b-07b8a29b89b8" containerID="e7edf1bc2baff758f75c9dfde54f53dd64e1ef5467a718ef5434592989042bc0" exitCode=0
Oct 30 22:55:46.858858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.858484    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v" event={"ID":"1c8f88a7-86d4-42eb-839b-07b8a29b89b8","Type":"ContainerDied","Data":"e7edf1bc2baff758f75c9dfde54f53dd64e1ef5467a718ef5434592989042bc0"}
Oct 30 22:55:46.859916 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.859898    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"
Oct 30 22:55:46.860045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.859912    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn" event={"ID":"871c8322-59be-4114-87b2-08d17b5d7acc","Type":"ContainerDied","Data":"c815c848bbe7cfe5308929c9c722846724a314f8f0af7e109c9021249b66dc03"}
Oct 30 22:55:46.860045 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:46.859948    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c815c848bbe7cfe5308929c9c722846724a314f8f0af7e109c9021249b66dc03"
Oct 30 22:55:48.006687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.006658    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:48.062565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.062537    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-xks-ca\") pod \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") "
Oct 30 22:55:48.062774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.062593    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-kube-api-access-4bjxj\" (UniqueName: \"kubernetes.io/projected/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-kube-api-access-4bjxj\") pod \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") "
Oct 30 22:55:48.062774 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.062661    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-openshift-ca\") pod \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\" (UID: \"1c8f88a7-86d4-42eb-839b-07b8a29b89b8\") "
Oct 30 22:55:48.063042 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.063019    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-openshift-ca" pod "1c8f88a7-86d4-42eb-839b-07b8a29b89b8" (UID: "1c8f88a7-86d4-42eb-839b-07b8a29b89b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:48.064654 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.064606    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-xks-ca" pod "1c8f88a7-86d4-42eb-839b-07b8a29b89b8" (UID: "1c8f88a7-86d4-42eb-839b-07b8a29b89b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:48.064768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.064680    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-kube-api-access-4bjxj" pod "1c8f88a7-86d4-42eb-839b-07b8a29b89b8" (UID: "1c8f88a7-86d4-42eb-839b-07b8a29b89b8"). InnerVolumeSpecName "kube-api-access-4bjxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:48.164211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.164184    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjxj\" (UniqueName: \"kubernetes.io/projected/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-kube-api-access-4bjxj\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:48.164211 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.164208    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:48.164414 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.164220    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c8f88a7-86d4-42eb-839b-07b8a29b89b8-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:48.873347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.873303    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v" event={"ID":"1c8f88a7-86d4-42eb-839b-07b8a29b89b8","Type":"ContainerDied","Data":"b88d2071f34619a176054ebf8ae7f3645acbd0f2df56704ad160e8559ca5cc0c"}
Oct 30 22:55:48.873347 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.873333    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"
Oct 30 22:55:48.873676 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.873360    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b88d2071f34619a176054ebf8ae7f3645acbd0f2df56704ad160e8559ca5cc0c"
Oct 30 22:55:48.900675 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.900630    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"]
Oct 30 22:55:48.947753 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.947699    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"]
Oct 30 22:55:48.949470 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:48.949445    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"]
Oct 30 22:55:49.082609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:49.082575    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:49.869029 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:49.868990    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:55:50.064308 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:50.064272    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"]
Oct 30 22:55:50.072671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:50.072609    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3adac6e5-4ck4g"]
Oct 30 22:55:50.095767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:50.095737    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"]
Oct 30 22:55:50.110998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:50.110968    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3812fa8d-r7z4k"]
Oct 30 22:55:50.160883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:50.160841    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:51.787728 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:51.787693    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334853a1-ed1a-44c4-8964-84a7f51c4ca8" path="/var/lib/kubelet/pods/334853a1-ed1a-44c4-8964-84a7f51c4ca8/volumes"
Oct 30 22:55:51.788154 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:51.788099    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="562effb4-26cd-462a-898f-3cb93e112042" path="/var/lib/kubelet/pods/562effb4-26cd-462a-898f-3cb93e112042/volumes"
Oct 30 22:55:54.114077 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:54.114046    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"]
Oct 30 22:55:54.515827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:54.515788    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:54.673646 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:54.673604    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:54.830251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:54.830121    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:57.128761 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:57.128725    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:57.282813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:57.282769    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:58.098398 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:55:58.098356    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:55:58.722353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.722320    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"]
Oct 30 22:55:58.722771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.722725    5503 cpu_manager.go:414] "containerMap: removing container" podUID="871c8322-59be-4114-87b2-08d17b5d7acc" containerName="revoke-keys"
Oct 30 22:55:58.722771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.722737    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871c8322-59be-4114-87b2-08d17b5d7acc" containerName="revoke-keys"
Oct 30 22:55:58.722771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.722751    5503 cpu_manager.go:414] "containerMap: removing container" podUID="1c8f88a7-86d4-42eb-839b-07b8a29b89b8" containerName="revoke-keys"
Oct 30 22:55:58.722771 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.722757    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8f88a7-86d4-42eb-839b-07b8a29b89b8" containerName="revoke-keys"
Oct 30 22:55:58.722920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.722822    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="871c8322-59be-4114-87b2-08d17b5d7acc" containerName="revoke-keys"
Oct 30 22:55:58.722920 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.722833    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8f88a7-86d4-42eb-839b-07b8a29b89b8" containerName="revoke-keys"
Oct 30 22:55:58.727059 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.727041    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.778729 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.778707    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:55:58.780888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.780856    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"]
Oct 30 22:55:58.825148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.825126    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:55:58.825148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.825136    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:55:58.825323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.825190    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:55:58.871800 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.871772    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnmn\" (UniqueName: \"kubernetes.io/projected/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-kube-api-access-4cnmn\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.871968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.871811    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.871968 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.871846    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.973203 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.973122    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.973375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.973239    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnmn\" (UniqueName: \"kubernetes.io/projected/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-kube-api-access-4cnmn\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.973375 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.973271    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.973814 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.973782    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:58.975604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:58.975583    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:59.015050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.015018    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"]
Oct 30 22:55:59.018624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.018604    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.031866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.031840    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnmn\" (UniqueName: \"kubernetes.io/projected/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-kube-api-access-4cnmn\") pod \"maas-api-revoke-sub-85818335-qrgwd\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:59.039535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.039514    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:55:59.073611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.073578    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"]
Oct 30 22:55:59.114713 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.114678    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf" podUID="25f6646e-5cf5-4da1-9ba3-70b826b59525" containerName="payload-processing" containerID="cri-o://6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15" gracePeriod=25
Oct 30 22:55:59.174449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.174418    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.174626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.174530    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2n42\" (UniqueName: \"kubernetes.io/projected/477f6424-6017-4c48-ab49-37d8111f5ba2-kube-api-access-k2n42\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.174626 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.174618    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.275822 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.275787    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2n42\" (UniqueName: \"kubernetes.io/projected/477f6424-6017-4c48-ab49-37d8111f5ba2-kube-api-access-k2n42\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.276024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.275888    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.276024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.275950    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.276721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.276692    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.278458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.278428    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.334671 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.334542    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2n42\" (UniqueName: \"kubernetes.io/projected/477f6424-6017-4c48-ab49-37d8111f5ba2-kube-api-access-k2n42\") pod \"maas-api-revoke-sub-4ef731d1-w9ngv\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.343374 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.343350    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:55:59.411464 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.411433    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"]
Oct 30 22:55:59.412535 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:55:59.412516    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f7b283_dc0a_49d3_ac55_8d316cc9d93f.slice/crio-46e66c68041958ac1688810b9d774717cf2e99cc9785738f86b03095c74a1e2f WatchSource:0}: Error finding container 46e66c68041958ac1688810b9d774717cf2e99cc9785738f86b03095c74a1e2f: Status 404 returned error can't find the container with id 46e66c68041958ac1688810b9d774717cf2e99cc9785738f86b03095c74a1e2f
Oct 30 22:55:59.477952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.477883    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f6646e-5cf5-4da1-9ba3-70b826b59525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f6646e-5cf5-4da1-9ba3-70b826b59525-plugins-config-volume\") pod \"25f6646e-5cf5-4da1-9ba3-70b826b59525\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") "
Oct 30 22:55:59.477952 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.477917    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f6646e-5cf5-4da1-9ba3-70b826b59525-kube-api-access-flfgq\" (UniqueName: \"kubernetes.io/projected/25f6646e-5cf5-4da1-9ba3-70b826b59525-kube-api-access-flfgq\") pod \"25f6646e-5cf5-4da1-9ba3-70b826b59525\" (UID: \"25f6646e-5cf5-4da1-9ba3-70b826b59525\") "
Oct 30 22:55:59.478298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.478275    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f6646e-5cf5-4da1-9ba3-70b826b59525-plugins-config-volume" pod "25f6646e-5cf5-4da1-9ba3-70b826b59525" (UID: "25f6646e-5cf5-4da1-9ba3-70b826b59525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:59.480014 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.479988    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f6646e-5cf5-4da1-9ba3-70b826b59525-kube-api-access-flfgq" pod "25f6646e-5cf5-4da1-9ba3-70b826b59525" (UID: "25f6646e-5cf5-4da1-9ba3-70b826b59525"). InnerVolumeSpecName "kube-api-access-flfgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:59.519805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.519771    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:55:59.579239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.579202    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25f6646e-5cf5-4da1-9ba3-70b826b59525-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:59.579239 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.579233    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flfgq\" (UniqueName: \"kubernetes.io/projected/25f6646e-5cf5-4da1-9ba3-70b826b59525-kube-api-access-flfgq\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:59.631337 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.631310    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:55:59.937079 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.937039    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd" event={"ID":"60f7b283-dc0a-49d3-ac55-8d316cc9d93f","Type":"ContainerStarted","Data":"73d46070795d94a515535aa91c5c2d31416e0164c72c0b8daf32852a55d281c7"}
Oct 30 22:55:59.937514 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.937105    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd" event={"ID":"60f7b283-dc0a-49d3-ac55-8d316cc9d93f","Type":"ContainerStarted","Data":"46e66c68041958ac1688810b9d774717cf2e99cc9785738f86b03095c74a1e2f"}
Oct 30 22:55:59.938455 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.938421    5503 generic.go:356] "Generic (PLEG): container finished" podID="25f6646e-5cf5-4da1-9ba3-70b826b59525" containerID="6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15" exitCode=0
Oct 30 22:55:59.938617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.938477    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"
Oct 30 22:55:59.938617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.938508    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf" event={"ID":"25f6646e-5cf5-4da1-9ba3-70b826b59525","Type":"ContainerDied","Data":"6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15"}
Oct 30 22:55:59.938617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.938566    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf" event={"ID":"25f6646e-5cf5-4da1-9ba3-70b826b59525","Type":"ContainerDied","Data":"01fb504681e9365acaa3d7a454e4c970d63b90e39a1d2742890ac24c751cf8b8"}
Oct 30 22:55:59.938617 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.938590    5503 scope.go:122] "RemoveContainer" containerID="6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15"
Oct 30 22:55:59.948038 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.948017    5503 scope.go:122] "RemoveContainer" containerID="6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15"
Oct 30 22:55:59.948303 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:55:59.948283    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15\": container with ID starting with 6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15 not found: ID does not exist" containerID="6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15"
Oct 30 22:55:59.948351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:55:59.948310    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15"} err="failed to get container status \"6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15\": rpc error: code = NotFound desc = could not find container \"6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15\": container with ID starting with 6d2f9333bb6d89df961be19ad557e14107f00a568f84b636e83d8d1e2ea2cc15 not found: ID does not exist"
Oct 30 22:56:00.001523 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.001499    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"]
Oct 30 22:56:00.003524 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:56:00.003495    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477f6424_6017_4c48_ab49_37d8111f5ba2.slice/crio-78d315dfa7c87aadc0969219ee5bfbc9b58d23c94dd03bade3495b9f767829b8 WatchSource:0}: Error finding container 78d315dfa7c87aadc0969219ee5bfbc9b58d23c94dd03bade3495b9f767829b8: Status 404 returned error can't find the container with id 78d315dfa7c87aadc0969219ee5bfbc9b58d23c94dd03bade3495b9f767829b8
Oct 30 22:56:00.048294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.048221    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd" podStartSLOduration=2.048207325 podStartE2EDuration="2.048207325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:00.047964197 +0000 UTC m=+4136.382540972" watchObservedRunningTime="2026-10-30 22:56:00.048207325 +0000 UTC m=+4136.382784095"
Oct 30 22:56:00.154861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.154824    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"]
Oct 30 22:56:00.159107 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.159076    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3b871a-7887df7cd9-8bpjf"]
Oct 30 22:56:00.948335 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.948301    5503 generic.go:356] "Generic (PLEG): container finished" podID="477f6424-6017-4c48-ab49-37d8111f5ba2" containerID="8f6858144d39e77f3f9d538f2a4fb05bc0c863a3bc843beb6a858895c2d2d660" exitCode=0
Oct 30 22:56:00.948823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.948397    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv" event={"ID":"477f6424-6017-4c48-ab49-37d8111f5ba2","Type":"ContainerDied","Data":"8f6858144d39e77f3f9d538f2a4fb05bc0c863a3bc843beb6a858895c2d2d660"}
Oct 30 22:56:00.948823 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.948454    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv" event={"ID":"477f6424-6017-4c48-ab49-37d8111f5ba2","Type":"ContainerStarted","Data":"78d315dfa7c87aadc0969219ee5bfbc9b58d23c94dd03bade3495b9f767829b8"}
Oct 30 22:56:00.949974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.949946    5503 generic.go:356] "Generic (PLEG): container finished" podID="60f7b283-dc0a-49d3-ac55-8d316cc9d93f" containerID="73d46070795d94a515535aa91c5c2d31416e0164c72c0b8daf32852a55d281c7" exitCode=0
Oct 30 22:56:00.950105 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:00.949997    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd" event={"ID":"60f7b283-dc0a-49d3-ac55-8d316cc9d93f","Type":"ContainerDied","Data":"73d46070795d94a515535aa91c5c2d31416e0164c72c0b8daf32852a55d281c7"}
Oct 30 22:56:01.787777 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:01.787746    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f6646e-5cf5-4da1-9ba3-70b826b59525" path="/var/lib/kubelet/pods/25f6646e-5cf5-4da1-9ba3-70b826b59525/volumes"
Oct 30 22:56:02.108138 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.108115    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:56:02.110182 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.110165    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:56:02.206498 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206461    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477f6424-6017-4c48-ab49-37d8111f5ba2-kube-api-access-k2n42\" (UniqueName: \"kubernetes.io/projected/477f6424-6017-4c48-ab49-37d8111f5ba2-kube-api-access-k2n42\") pod \"477f6424-6017-4c48-ab49-37d8111f5ba2\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") "
Oct 30 22:56:02.206717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206519    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-openshift-ca\") pod \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") "
Oct 30 22:56:02.206717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206557    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-kube-api-access-4cnmn\" (UniqueName: \"kubernetes.io/projected/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-kube-api-access-4cnmn\") pod \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") "
Oct 30 22:56:02.206717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206587    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-xks-ca\") pod \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\" (UID: \"60f7b283-dc0a-49d3-ac55-8d316cc9d93f\") "
Oct 30 22:56:02.206717 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206604    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-openshift-ca\") pod \"477f6424-6017-4c48-ab49-37d8111f5ba2\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") "
Oct 30 22:56:02.206953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206801    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-xks-ca\") pod \"477f6424-6017-4c48-ab49-37d8111f5ba2\" (UID: \"477f6424-6017-4c48-ab49-37d8111f5ba2\") "
Oct 30 22:56:02.207004 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206954    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-openshift-ca" pod "60f7b283-dc0a-49d3-ac55-8d316cc9d93f" (UID: "60f7b283-dc0a-49d3-ac55-8d316cc9d93f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:02.207054 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.206992    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-openshift-ca" pod "477f6424-6017-4c48-ab49-37d8111f5ba2" (UID: "477f6424-6017-4c48-ab49-37d8111f5ba2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:02.207285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.207261    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:02.207403 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.207287    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:02.208864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.208827    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-xks-ca" pod "60f7b283-dc0a-49d3-ac55-8d316cc9d93f" (UID: "60f7b283-dc0a-49d3-ac55-8d316cc9d93f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:02.208864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.208856    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477f6424-6017-4c48-ab49-37d8111f5ba2-kube-api-access-k2n42" pod "477f6424-6017-4c48-ab49-37d8111f5ba2" (UID: "477f6424-6017-4c48-ab49-37d8111f5ba2"). InnerVolumeSpecName "kube-api-access-k2n42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:02.208998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.208868    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-xks-ca" pod "477f6424-6017-4c48-ab49-37d8111f5ba2" (UID: "477f6424-6017-4c48-ab49-37d8111f5ba2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:02.208998 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.208915    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-kube-api-access-4cnmn" pod "60f7b283-dc0a-49d3-ac55-8d316cc9d93f" (UID: "60f7b283-dc0a-49d3-ac55-8d316cc9d93f"). InnerVolumeSpecName "kube-api-access-4cnmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:02.308552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.308519    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2n42\" (UniqueName: \"kubernetes.io/projected/477f6424-6017-4c48-ab49-37d8111f5ba2-kube-api-access-k2n42\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:02.308552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.308547    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnmn\" (UniqueName: \"kubernetes.io/projected/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-kube-api-access-4cnmn\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:02.308552 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.308561    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f7b283-dc0a-49d3-ac55-8d316cc9d93f-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:02.308843 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.308574    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/477f6424-6017-4c48-ab49-37d8111f5ba2-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:02.963611 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.963578    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv"
Oct 30 22:56:02.963840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.963602    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef731d1-w9ngv" event={"ID":"477f6424-6017-4c48-ab49-37d8111f5ba2","Type":"ContainerDied","Data":"78d315dfa7c87aadc0969219ee5bfbc9b58d23c94dd03bade3495b9f767829b8"}
Oct 30 22:56:02.963840 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.963681    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d315dfa7c87aadc0969219ee5bfbc9b58d23c94dd03bade3495b9f767829b8"
Oct 30 22:56:02.965065 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.965042    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd"
Oct 30 22:56:02.965233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.965081    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85818335-qrgwd" event={"ID":"60f7b283-dc0a-49d3-ac55-8d316cc9d93f","Type":"ContainerDied","Data":"46e66c68041958ac1688810b9d774717cf2e99cc9785738f86b03095c74a1e2f"}
Oct 30 22:56:02.965233 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:02.965117    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e66c68041958ac1688810b9d774717cf2e99cc9785738f86b03095c74a1e2f"
Oct 30 22:56:03.613364 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:03.613327    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:03.921282 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:03.921245    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv" podUID="8bc93625-5e4f-4e54-b50c-87731dc961e6" containerName="main" containerID="cri-o://1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f" gracePeriod=45
Oct 30 22:56:03.966913 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:03.966853    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main" containerID="cri-o://a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6" gracePeriod=45
Oct 30 22:56:03.968343 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:03.968291    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main" containerID="cri-o://3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa" gracePeriod=45
Oct 30 22:56:04.247109 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.247082    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:56:04.261000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.260951    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:56:04.264033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.264014    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:56:04.432666 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432623    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-dshm\") pod \"e2e93339-64a5-4beb-a491-557f5d4506e0\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") "
Oct 30 22:56:04.432865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432681    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-model-cache\") pod \"8bc93625-5e4f-4e54-b50c-87731dc961e6\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") "
Oct 30 22:56:04.432865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432705    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-home\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-home\") pod \"8bc93625-5e4f-4e54-b50c-87731dc961e6\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") "
Oct 30 22:56:04.432865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432747    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tls-certs\" (UniqueName: \"kubernetes.io/secret/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tls-certs\") pod \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") "
Oct 30 22:56:04.432865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432787    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-model-cache\") pod \"e2e93339-64a5-4beb-a491-557f5d4506e0\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") "
Oct 30 22:56:04.432865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432814    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e93339-64a5-4beb-a491-557f5d4506e0-kube-api-access-74cn9\" (UniqueName: \"kubernetes.io/projected/e2e93339-64a5-4beb-a491-557f5d4506e0-kube-api-access-74cn9\") pod \"e2e93339-64a5-4beb-a491-557f5d4506e0\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") "
Oct 30 22:56:04.432865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432833    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e93339-64a5-4beb-a491-557f5d4506e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e93339-64a5-4beb-a491-557f5d4506e0-tls-certs\") pod \"e2e93339-64a5-4beb-a491-557f5d4506e0\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") "
Oct 30 22:56:04.432865 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432851    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-dshm\") pod \"8bc93625-5e4f-4e54-b50c-87731dc961e6\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432903    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-dshm\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-dshm\") pod \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432941    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-home\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-home\") pod \"e2e93339-64a5-4beb-a491-557f5d4506e0\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.432972    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-kube-api-access-qpkl8\" (UniqueName: \"kubernetes.io/projected/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-kube-api-access-qpkl8\") pod \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433007    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-tmp-dir\") pod \"e2e93339-64a5-4beb-a491-557f5d4506e0\" (UID: \"e2e93339-64a5-4beb-a491-557f5d4506e0\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433031    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-home" pod "8bc93625-5e4f-4e54-b50c-87731dc961e6" (UID: "8bc93625-5e4f-4e54-b50c-87731dc961e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433025    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-model-cache" pod "8bc93625-5e4f-4e54-b50c-87731dc961e6" (UID: "8bc93625-5e4f-4e54-b50c-87731dc961e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433039    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc93625-5e4f-4e54-b50c-87731dc961e6-kube-api-access-8t77z\" (UniqueName: \"kubernetes.io/projected/8bc93625-5e4f-4e54-b50c-87731dc961e6-kube-api-access-8t77z\") pod \"8bc93625-5e4f-4e54-b50c-87731dc961e6\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433119    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-home\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-home\") pod \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433187    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tmp-dir\") pod \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") "
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433225    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-model-cache" pod "e2e93339-64a5-4beb-a491-557f5d4506e0" (UID: "e2e93339-64a5-4beb-a491-557f5d4506e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.433273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433228    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-model-cache\") pod \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\" (UID: \"954db4d4-4b0e-461f-aa6c-9b71d2ed6340\") "
Oct 30 22:56:04.433930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433346    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-tmp-dir\") pod \"8bc93625-5e4f-4e54-b50c-87731dc961e6\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") "
Oct 30 22:56:04.433930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433380    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc93625-5e4f-4e54-b50c-87731dc961e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc93625-5e4f-4e54-b50c-87731dc961e6-tls-certs\") pod \"8bc93625-5e4f-4e54-b50c-87731dc961e6\" (UID: \"8bc93625-5e4f-4e54-b50c-87731dc961e6\") "
Oct 30 22:56:04.433930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433388    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-home" pod "e2e93339-64a5-4beb-a491-557f5d4506e0" (UID: "e2e93339-64a5-4beb-a491-557f5d4506e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.433930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433397    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-model-cache" pod "954db4d4-4b0e-461f-aa6c-9b71d2ed6340" (UID: "954db4d4-4b0e-461f-aa6c-9b71d2ed6340"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.433930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433571    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-home" pod "954db4d4-4b0e-461f-aa6c-9b71d2ed6340" (UID: "954db4d4-4b0e-461f-aa6c-9b71d2ed6340"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.433930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433738    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tmp-dir" pod "954db4d4-4b0e-461f-aa6c-9b71d2ed6340" (UID: "954db4d4-4b0e-461f-aa6c-9b71d2ed6340"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.433930 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433917    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433939    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433951    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433963    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433991    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.434004    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.434018    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.433956    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-tmp-dir" pod "e2e93339-64a5-4beb-a491-557f5d4506e0" (UID: "e2e93339-64a5-4beb-a491-557f5d4506e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.434312 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.434009    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-tmp-dir" pod "8bc93625-5e4f-4e54-b50c-87731dc961e6" (UID: "8bc93625-5e4f-4e54-b50c-87731dc961e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.435724 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.435685    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-dshm" pod "e2e93339-64a5-4beb-a491-557f5d4506e0" (UID: "e2e93339-64a5-4beb-a491-557f5d4506e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.435857 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.435826    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e93339-64a5-4beb-a491-557f5d4506e0-kube-api-access-74cn9" pod "e2e93339-64a5-4beb-a491-557f5d4506e0" (UID: "e2e93339-64a5-4beb-a491-557f5d4506e0"). InnerVolumeSpecName "kube-api-access-74cn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:04.436245 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.436199    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-dshm" pod "8bc93625-5e4f-4e54-b50c-87731dc961e6" (UID: "8bc93625-5e4f-4e54-b50c-87731dc961e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.436378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.436303    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tls-certs" pod "954db4d4-4b0e-461f-aa6c-9b71d2ed6340" (UID: "954db4d4-4b0e-461f-aa6c-9b71d2ed6340"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:04.436378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.436326    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc93625-5e4f-4e54-b50c-87731dc961e6-tls-certs" pod "8bc93625-5e4f-4e54-b50c-87731dc961e6" (UID: "8bc93625-5e4f-4e54-b50c-87731dc961e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:04.436378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.436303    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e93339-64a5-4beb-a491-557f5d4506e0-tls-certs" pod "e2e93339-64a5-4beb-a491-557f5d4506e0" (UID: "e2e93339-64a5-4beb-a491-557f5d4506e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:04.436378 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.436363    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc93625-5e4f-4e54-b50c-87731dc961e6-kube-api-access-8t77z" pod "8bc93625-5e4f-4e54-b50c-87731dc961e6" (UID: "8bc93625-5e4f-4e54-b50c-87731dc961e6"). InnerVolumeSpecName "kube-api-access-8t77z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:04.436656 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.436620    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-kube-api-access-qpkl8" pod "954db4d4-4b0e-461f-aa6c-9b71d2ed6340" (UID: "954db4d4-4b0e-461f-aa6c-9b71d2ed6340"). InnerVolumeSpecName "kube-api-access-qpkl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:04.437219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.437199    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-dshm" pod "954db4d4-4b0e-461f-aa6c-9b71d2ed6340" (UID: "954db4d4-4b0e-461f-aa6c-9b71d2ed6340"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:04.534978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534902    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.534978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534929    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bc93625-5e4f-4e54-b50c-87731dc961e6-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.534978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534938    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.534978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534946    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.534978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534955    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74cn9\" (UniqueName: \"kubernetes.io/projected/e2e93339-64a5-4beb-a491-557f5d4506e0-kube-api-access-74cn9\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.534978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534964    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2e93339-64a5-4beb-a491-557f5d4506e0-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.534978 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534973    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bc93625-5e4f-4e54-b50c-87731dc961e6-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.535323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534984    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.535323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.534994    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkl8\" (UniqueName: \"kubernetes.io/projected/954db4d4-4b0e-461f-aa6c-9b71d2ed6340-kube-api-access-qpkl8\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.535323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.535004    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2e93339-64a5-4beb-a491-557f5d4506e0-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.535323 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.535012    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t77z\" (UniqueName: \"kubernetes.io/projected/8bc93625-5e4f-4e54-b50c-87731dc961e6-kube-api-access-8t77z\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:04.979426 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.979396    5503 generic.go:356] "Generic (PLEG): container finished" podID="8bc93625-5e4f-4e54-b50c-87731dc961e6" containerID="1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f" exitCode=0
Oct 30 22:56:04.979842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.979445    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"
Oct 30 22:56:04.979842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.979493    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv" event={"ID":"8bc93625-5e4f-4e54-b50c-87731dc961e6","Type":"ContainerDied","Data":"1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f"}
Oct 30 22:56:04.979842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.979537    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv" event={"ID":"8bc93625-5e4f-4e54-b50c-87731dc961e6","Type":"ContainerDied","Data":"288a602bca0cc062dcfe123caa35c968711126061fa04fb99e28dc8b954ffe83"}
Oct 30 22:56:04.979842 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.979553    5503 scope.go:122] "RemoveContainer" containerID="1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f"
Oct 30 22:56:04.981089 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.981068    5503 generic.go:356] "Generic (PLEG): container finished" podID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerID="a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6" exitCode=0
Oct 30 22:56:04.981215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.981101    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" event={"ID":"e2e93339-64a5-4beb-a491-557f5d4506e0","Type":"ContainerDied","Data":"a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6"}
Oct 30 22:56:04.981215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.981118    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"
Oct 30 22:56:04.981215 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.981144    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8" event={"ID":"e2e93339-64a5-4beb-a491-557f5d4506e0","Type":"ContainerDied","Data":"5cc032441c7bb3cb5f16b0e8c28d7ac953b5062bcc7d278cfb24c28bfc6575da"}
Oct 30 22:56:04.982457 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.982434    5503 generic.go:356] "Generic (PLEG): container finished" podID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerID="3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa" exitCode=0
Oct 30 22:56:04.982565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.982457    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" event={"ID":"954db4d4-4b0e-461f-aa6c-9b71d2ed6340","Type":"ContainerDied","Data":"3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa"}
Oct 30 22:56:04.982565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.982472    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"
Oct 30 22:56:04.982565 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.982495    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z" event={"ID":"954db4d4-4b0e-461f-aa6c-9b71d2ed6340","Type":"ContainerDied","Data":"d4fdf67a1fb2a758f1626e122ebed5e195f82959b7a2393b2fbc0c69fadc3cfe"}
Oct 30 22:56:04.991110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.991095    5503 scope.go:122] "RemoveContainer" containerID="1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f"
Oct 30 22:56:04.991365 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:56:04.991347    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f\": container with ID starting with 1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f not found: ID does not exist" containerID="1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f"
Oct 30 22:56:04.991416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.991373    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f"} err="failed to get container status \"1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f\": rpc error: code = NotFound desc = could not find container \"1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f\": container with ID starting with 1cf83992f7e0304e18fce01b1ff2f49b7ef80ad74774a793bf10d0cf0734884f not found: ID does not exist"
Oct 30 22:56:04.991416 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:04.991390    5503 scope.go:122] "RemoveContainer" containerID="a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6"
Oct 30 22:56:05.000104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.000081    5503 scope.go:122] "RemoveContainer" containerID="a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6"
Oct 30 22:56:05.000384 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:56:05.000364    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6\": container with ID starting with a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6 not found: ID does not exist" containerID="a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6"
Oct 30 22:56:05.000450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.000391    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6"} err="failed to get container status \"a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6\": rpc error: code = NotFound desc = could not find container \"a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6\": container with ID starting with a09c7af819fcad23b57168e6d91c30385bb5d80e9245deb28d964110b86892a6 not found: ID does not exist"
Oct 30 22:56:05.000450 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.000409    5503 scope.go:122] "RemoveContainer" containerID="3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa"
Oct 30 22:56:05.008442 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.008426    5503 scope.go:122] "RemoveContainer" containerID="3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa"
Oct 30 22:56:05.008699 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:56:05.008677    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa\": container with ID starting with 3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa not found: ID does not exist" containerID="3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa"
Oct 30 22:56:05.008764 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.008713    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa"} err="failed to get container status \"3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa\": rpc error: code = NotFound desc = could not find container \"3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa\": container with ID starting with 3b96093840223fdfaf2ebc20fa1ef9987986601bce32da1b9a2fdf8a4e70b0aa not found: ID does not exist"
Oct 30 22:56:05.107454 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.107422    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:05.135293 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.135256    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"]
Oct 30 22:56:05.142174 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.142149    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/simulator-w2-7f66ab-kserve-596dcf649f-nft4z"]
Oct 30 22:56:05.239961 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.239864    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv" podUID="8bc93625-5e4f-4e54-b50c-87731dc961e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:56:05.256111 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.256078    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"]
Oct 30 22:56:05.267351 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.267323    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/premium-w2-7f66ab-kserve-5b9884c8db-qgnc8"]
Oct 30 22:56:05.374953 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.374916    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"]
Oct 30 22:56:05.379485 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.379460    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7f66ab/distinct-2-w2-7f66ab-kserve-7c76d6f66b-xn9gv"]
Oct 30 22:56:05.413664 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.413477    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:05.580108 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.580026    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"]
Oct 30 22:56:05.584370 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.584345    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-3b871a-jjwwn"]
Oct 30 22:56:05.592120 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.592097    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:05.789051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.789019    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871c8322-59be-4114-87b2-08d17b5d7acc" path="/var/lib/kubelet/pods/871c8322-59be-4114-87b2-08d17b5d7acc/volumes"
Oct 30 22:56:05.789422 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.789409    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc93625-5e4f-4e54-b50c-87731dc961e6" path="/var/lib/kubelet/pods/8bc93625-5e4f-4e54-b50c-87731dc961e6/volumes"
Oct 30 22:56:05.789781 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.789768    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" path="/var/lib/kubelet/pods/954db4d4-4b0e-461f-aa6c-9b71d2ed6340/volumes"
Oct 30 22:56:05.790405 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:05.790389    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" path="/var/lib/kubelet/pods/e2e93339-64a5-4beb-a491-557f5d4506e0/volumes"
Oct 30 22:56:06.032884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:06.032849    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:06.286050 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:06.285958    5503 scope.go:122] "RemoveContainer" containerID="e5648a94ff890ca99eda9c8db232dc192206b0dc795d747460688e3ecc13fa4f"
Oct 30 22:56:06.295206 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:06.295185    5503 scope.go:122] "RemoveContainer" containerID="64c18d1dd969388dd6caada9a43ba6a49a08976e9d92e1415a70f3689a0e5bf3"
Oct 30 22:56:06.304013 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:06.303996    5503 scope.go:122] "RemoveContainer" containerID="1c76427e8a76d5cc927603fb3defaf75a85c6973f7c68af898b97c242c9aae01"
Oct 30 22:56:06.312479 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:06.312459    5503 scope.go:122] "RemoveContainer" containerID="b491c817225a57a9ac270aec660254bc6858c7fcdc5db93441cc96c1d334398b"
Oct 30 22:56:10.137986 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:10.137947    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:13.505999 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:13.505903    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:13.676588 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:13.676554    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:13.931625 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:13.931592    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:14.764936 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.764904    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"]
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765256    5503 cpu_manager.go:414] "containerMap: removing container" podUID="25f6646e-5cf5-4da1-9ba3-70b826b59525" containerName="payload-processing"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765267    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f6646e-5cf5-4da1-9ba3-70b826b59525" containerName="payload-processing"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765280    5503 cpu_manager.go:414] "containerMap: removing container" podUID="60f7b283-dc0a-49d3-ac55-8d316cc9d93f" containerName="revoke-keys"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765285    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f7b283-dc0a-49d3-ac55-8d316cc9d93f" containerName="revoke-keys"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765292    5503 cpu_manager.go:414] "containerMap: removing container" podUID="477f6424-6017-4c48-ab49-37d8111f5ba2" containerName="revoke-keys"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765297    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477f6424-6017-4c48-ab49-37d8111f5ba2" containerName="revoke-keys"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765316    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765322    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765349    5503 cpu_manager.go:414] "containerMap: removing container" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main"
Oct 30 22:56:14.765353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765353    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765361    5503 cpu_manager.go:414] "containerMap: removing container" podUID="8bc93625-5e4f-4e54-b50c-87731dc961e6" containerName="main"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765365    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc93625-5e4f-4e54-b50c-87731dc961e6" containerName="main"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765419    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f6646e-5cf5-4da1-9ba3-70b826b59525" containerName="payload-processing"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765429    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f7b283-dc0a-49d3-ac55-8d316cc9d93f" containerName="revoke-keys"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765434    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="477f6424-6017-4c48-ab49-37d8111f5ba2" containerName="revoke-keys"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765439    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e93339-64a5-4beb-a491-557f5d4506e0" containerName="main"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765449    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="954db4d4-4b0e-461f-aa6c-9b71d2ed6340" containerName="main"
Oct 30 22:56:14.765718 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.765455    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc93625-5e4f-4e54-b50c-87731dc961e6" containerName="main"
Oct 30 22:56:14.768489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.768471    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.818170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.818132    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:14.818170 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.818164    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:14.822291 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.822258    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbfpf\" (UniqueName: \"kubernetes.io/projected/ce373d90-e7b8-4d63-a43e-c7e706339268-kube-api-access-hbfpf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.822445 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.822321    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.822522 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.822461    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.865011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.864982    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:14.865011 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.865007    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:56:14.866992 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.866973    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"]
Oct 30 22:56:14.905754 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.905720    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:14.923400 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.923369    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfpf\" (UniqueName: \"kubernetes.io/projected/ce373d90-e7b8-4d63-a43e-c7e706339268-kube-api-access-hbfpf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.923574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.923415    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.923574 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.923459    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.924067 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.924047    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.925841 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.925821    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:14.984483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:14.984446    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfpf\" (UniqueName: \"kubernetes.io/projected/ce373d90-e7b8-4d63-a43e-c7e706339268-kube-api-access-hbfpf\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:15.082196 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:15.082119    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:15.447258 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:15.447232    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"]
Oct 30 22:56:16.052809 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:16.052769    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s" event={"ID":"ce373d90-e7b8-4d63-a43e-c7e706339268","Type":"ContainerStarted","Data":"f930d911031fd2e8ea02b4aeedd5c8af308594f2bf33c9615e3df7f33dad5965"}
Oct 30 22:56:16.053209 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:16.052820    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s" event={"ID":"ce373d90-e7b8-4d63-a43e-c7e706339268","Type":"ContainerStarted","Data":"d160ad93448b186ef2917de3e146e9d015c07b416ebdbc15ad3294d7fef84169"}
Oct 30 22:56:16.164540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:16.164484    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s" podStartSLOduration=2.164468326 podStartE2EDuration="2.164468326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:56:16.16302555 +0000 UTC m=+4152.497602344" watchObservedRunningTime="2026-10-30 22:56:16.164468326 +0000 UTC m=+4152.499045094"
Oct 30 22:56:17.059824 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:17.059783    5503 generic.go:356] "Generic (PLEG): container finished" podID="ce373d90-e7b8-4d63-a43e-c7e706339268" containerID="f930d911031fd2e8ea02b4aeedd5c8af308594f2bf33c9615e3df7f33dad5965" exitCode=0
Oct 30 22:56:17.060197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:17.059838    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s" event={"ID":"ce373d90-e7b8-4d63-a43e-c7e706339268","Type":"ContainerDied","Data":"f930d911031fd2e8ea02b4aeedd5c8af308594f2bf33c9615e3df7f33dad5965"}
Oct 30 22:56:18.196663 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.196622    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:18.254581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.254550    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce373d90-e7b8-4d63-a43e-c7e706339268-kube-api-access-hbfpf\" (UniqueName: \"kubernetes.io/projected/ce373d90-e7b8-4d63-a43e-c7e706339268-kube-api-access-hbfpf\") pod \"ce373d90-e7b8-4d63-a43e-c7e706339268\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") "
Oct 30 22:56:18.254758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.254622    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-openshift-ca\") pod \"ce373d90-e7b8-4d63-a43e-c7e706339268\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") "
Oct 30 22:56:18.254758 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.254723    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-xks-ca\") pod \"ce373d90-e7b8-4d63-a43e-c7e706339268\" (UID: \"ce373d90-e7b8-4d63-a43e-c7e706339268\") "
Oct 30 22:56:18.255001 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.254978    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-openshift-ca" pod "ce373d90-e7b8-4d63-a43e-c7e706339268" (UID: "ce373d90-e7b8-4d63-a43e-c7e706339268"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:18.255085 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.255073    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:18.256739 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.256713    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce373d90-e7b8-4d63-a43e-c7e706339268-kube-api-access-hbfpf" pod "ce373d90-e7b8-4d63-a43e-c7e706339268" (UID: "ce373d90-e7b8-4d63-a43e-c7e706339268"). InnerVolumeSpecName "kube-api-access-hbfpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:18.256835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.256796    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-xks-ca" pod "ce373d90-e7b8-4d63-a43e-c7e706339268" (UID: "ce373d90-e7b8-4d63-a43e-c7e706339268"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:18.355864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.355781    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbfpf\" (UniqueName: \"kubernetes.io/projected/ce373d90-e7b8-4d63-a43e-c7e706339268-kube-api-access-hbfpf\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:18.355864 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:18.355812    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce373d90-e7b8-4d63-a43e-c7e706339268-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:19.072326 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:19.072296    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"
Oct 30 22:56:19.072487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:19.072328    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s" event={"ID":"ce373d90-e7b8-4d63-a43e-c7e706339268","Type":"ContainerDied","Data":"d160ad93448b186ef2917de3e146e9d015c07b416ebdbc15ad3294d7fef84169"}
Oct 30 22:56:19.072487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:19.072372    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d160ad93448b186ef2917de3e146e9d015c07b416ebdbc15ad3294d7fef84169"
Oct 30 22:56:19.734463 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:19.734432    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:20.961955 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:20.961915    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:56:23.749165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:23.749128    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:25.244140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:25.244102    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:25.417861 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:25.417828    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:25.579286 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:25.579197    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:25.720854 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:25.720822    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"]
Oct 30 22:56:25.767747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:25.767718    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-7f66ab-xvn9v"]
Oct 30 22:56:25.787768 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:25.787736    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8f88a7-86d4-42eb-839b-07b8a29b89b8" path="/var/lib/kubelet/pods/1c8f88a7-86d4-42eb-839b-07b8a29b89b8/volumes"
Oct 30 22:56:26.197119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:26.197081    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:26.370321 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:26.370288    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:27.712934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:27.712894    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:27.865487 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:27.865448    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:28.098242 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:56:28.098165    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:56:28.268812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:28.268780    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:29.631547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:29.631516    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:31.421070 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:31.421034    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:32.749599 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:32.749564    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:33.053721 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:33.053613    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"]
Oct 30 22:56:33.073590 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:33.073559    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf0bc5d-p86dc"]
Oct 30 22:56:33.672462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:33.672431    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:33.789285 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:33.789248    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405a1f97-068c-4506-b99a-d8a905f64cc7" path="/var/lib/kubelet/pods/405a1f97-068c-4506-b99a-d8a905f64cc7/volumes"
Oct 30 22:56:34.605734 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:34.605698    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:34.855229 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:34.855196    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:35.937361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:35.937327    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:40.523974 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:40.523933    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:45.045951 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:45.045917    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"]
Oct 30 22:56:45.052984 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:45.052955    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb029b16-7cdjk"]
Oct 30 22:56:45.787872 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:45.787835    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8939bcab-c8de-4e8e-b6d7-0035be050fb6" path="/var/lib/kubelet/pods/8939bcab-c8de-4e8e-b6d7-0035be050fb6/volumes"
Oct 30 22:56:45.861943 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:45.861912    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:45.984260 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:45.984218    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:56:46.125889 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:46.125803    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:46.280187 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:46.280152    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:46.313964 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:46.313922    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"]
Oct 30 22:56:46.360747 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:46.360711    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3b392d-9c22s"]
Oct 30 22:56:46.804049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:46.804013    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:47.788230 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:47.788195    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce373d90-e7b8-4d63-a43e-c7e706339268" path="/var/lib/kubelet/pods/ce373d90-e7b8-4d63-a43e-c7e706339268/volumes"
Oct 30 22:56:50.919471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:50.919434    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:53.210906 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:53.210871    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:53.374361 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:53.374330    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:53.588294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:53.588215    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:54.465165 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.465130    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"]
Oct 30 22:56:54.465575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.465498    5503 cpu_manager.go:414] "containerMap: removing container" podUID="ce373d90-e7b8-4d63-a43e-c7e706339268" containerName="revoke-keys"
Oct 30 22:56:54.465575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.465509    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce373d90-e7b8-4d63-a43e-c7e706339268" containerName="revoke-keys"
Oct 30 22:56:54.465687 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.465588    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce373d90-e7b8-4d63-a43e-c7e706339268" containerName="revoke-keys"
Oct 30 22:56:54.469845 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.469828    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.509032 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.508995    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.509199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.509081    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fsph\" (UniqueName: \"kubernetes.io/projected/25f9d380-01af-437b-ad7f-dd90f4f9e079-kube-api-access-6fsph\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.509199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.509169    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.527755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.527728    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:56:54.528003 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.527983    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:54.528049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.527988    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:56:54.574767 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.574738    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:56:54.576569 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.576551    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"]
Oct 30 22:56:54.610322 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.610294    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.610460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.610341    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fsph\" (UniqueName: \"kubernetes.io/projected/25f9d380-01af-437b-ad7f-dd90f4f9e079-kube-api-access-6fsph\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.610460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.610395    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.611114 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.611097    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.612707 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.612688    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.669356 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.669321    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fsph\" (UniqueName: \"kubernetes.io/projected/25f9d380-01af-437b-ad7f-dd90f4f9e079-kube-api-access-6fsph\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:54.782136 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:54.782057    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:55.149928 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:55.149900    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"]
Oct 30 22:56:55.152298 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:56:55.152266    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f9d380_01af_437b_ad7f_dd90f4f9e079.slice/crio-5a6e42a6638159e8001ffc62855cc716bcc5e3a290f02f1e9a849aafc757bad1 WatchSource:0}: Error finding container 5a6e42a6638159e8001ffc62855cc716bcc5e3a290f02f1e9a849aafc757bad1: Status 404 returned error can't find the container with id 5a6e42a6638159e8001ffc62855cc716bcc5e3a290f02f1e9a849aafc757bad1
Oct 30 22:56:55.282460 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:55.282417    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb" event={"ID":"25f9d380-01af-437b-ad7f-dd90f4f9e079","Type":"ContainerStarted","Data":"5a6e42a6638159e8001ffc62855cc716bcc5e3a290f02f1e9a849aafc757bad1"}
Oct 30 22:56:55.753709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:55.753605    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:56:56.289813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:56.289779    5503 generic.go:356] "Generic (PLEG): container finished" podID="25f9d380-01af-437b-ad7f-dd90f4f9e079" containerID="2d368b08a70893b863bf7282c113b180c90ee5f5d4f23d4bbd0e6a0bb8acd214" exitCode=0
Oct 30 22:56:56.290000 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:56.289846    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb" event={"ID":"25f9d380-01af-437b-ad7f-dd90f4f9e079","Type":"ContainerDied","Data":"2d368b08a70893b863bf7282c113b180c90ee5f5d4f23d4bbd0e6a0bb8acd214"}
Oct 30 22:56:57.429938 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.429910    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:57.434532 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.434515    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25f9d380-01af-437b-ad7f-dd90f4f9e079-kube-api-access-6fsph\" (UniqueName: \"kubernetes.io/projected/25f9d380-01af-437b-ad7f-dd90f4f9e079-kube-api-access-6fsph\") pod \"25f9d380-01af-437b-ad7f-dd90f4f9e079\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") "
Oct 30 22:56:57.434621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.434573    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-openshift-ca\") pod \"25f9d380-01af-437b-ad7f-dd90f4f9e079\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") "
Oct 30 22:56:57.434621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.434596    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-xks-ca\") pod \"25f9d380-01af-437b-ad7f-dd90f4f9e079\" (UID: \"25f9d380-01af-437b-ad7f-dd90f4f9e079\") "
Oct 30 22:56:57.434911 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.434890    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-openshift-ca" pod "25f9d380-01af-437b-ad7f-dd90f4f9e079" (UID: "25f9d380-01af-437b-ad7f-dd90f4f9e079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:57.436583 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.436558    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-xks-ca" pod "25f9d380-01af-437b-ad7f-dd90f4f9e079" (UID: "25f9d380-01af-437b-ad7f-dd90f4f9e079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:57.436727 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.436665    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25f9d380-01af-437b-ad7f-dd90f4f9e079-kube-api-access-6fsph" pod "25f9d380-01af-437b-ad7f-dd90f4f9e079" (UID: "25f9d380-01af-437b-ad7f-dd90f4f9e079"). InnerVolumeSpecName "kube-api-access-6fsph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:57.536088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.536055    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:57.536088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.536085    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25f9d380-01af-437b-ad7f-dd90f4f9e079-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:57.536088 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:57.536095    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fsph\" (UniqueName: \"kubernetes.io/projected/25f9d380-01af-437b-ad7f-dd90f4f9e079-kube-api-access-6fsph\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:58.098168 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:56:58.098084    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:56:58.302225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:58.302198    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"
Oct 30 22:56:58.302397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:58.302293    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb" event={"ID":"25f9d380-01af-437b-ad7f-dd90f4f9e079","Type":"ContainerDied","Data":"5a6e42a6638159e8001ffc62855cc716bcc5e3a290f02f1e9a849aafc757bad1"}
Oct 30 22:56:58.302397 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:56:58.302333    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a6e42a6638159e8001ffc62855cc716bcc5e3a290f02f1e9a849aafc757bad1"
Oct 30 22:57:00.415604 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:00.415564    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:03.046812 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:03.046781    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"]
Oct 30 22:57:03.093453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:03.093427    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2faed267-9m62d"]
Oct 30 22:57:03.787353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:03.787322    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d45169-b3f3-41b1-81a8-8dc4a0455942" path="/var/lib/kubelet/pods/56d45169-b3f3-41b1-81a8-8dc4a0455942/volumes"
Oct 30 22:57:04.474207 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:04.474178    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:57:04.486419 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:04.486387    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gvxg2_e5c7e5a7-6225-4513-a2cc-592dfe01d018/ovn-acl-logging/0.log"
Oct 30 22:57:04.926281 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:04.926246    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:06.382381 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:06.382351    5503 scope.go:122] "RemoveContainer" containerID="84d3e49a9a1cbb1143efea221bae14006c65adf14e7c6a0c3ba271b49bc14f6c"
Oct 30 22:57:06.391122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:06.391097    5503 scope.go:122] "RemoveContainer" containerID="4181c316b3d60f3662b4b3d591f88a1f2b8bf05e4faf1d4b45a13596db38ffd2"
Oct 30 22:57:06.409265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:06.409241    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:06.567385 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:06.567331    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:09.522294 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:09.522261    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:13.537430 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:13.537395    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:14.967458 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:14.967424    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:15.220873 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:15.220795    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:15.374969 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:15.374931    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:15.422568 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:15.422539    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"]
Oct 30 22:57:15.469535 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:15.469502    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-74bb81-hxzzb"]
Oct 30 22:57:15.787310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:15.787274    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25f9d380-01af-437b-ad7f-dd90f4f9e079" path="/var/lib/kubelet/pods/25f9d380-01af-437b-ad7f-dd90f4f9e079/volumes"
Oct 30 22:57:15.901596 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:15.901564    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:19.208511 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:19.208474    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:57:20.126407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:20.126379    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:22.104696 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:22.104659    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:22.267104 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:22.267062    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:22.443122 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:22.443093    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"]
Oct 30 22:57:22.443338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:22.443312    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy" containerID="cri-o://8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d" gracePeriod=30
Oct 30 22:57:22.460576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:22.460549    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:22.954866 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:22.954828    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:24.884333 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:24.884292    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:27.689117 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.689092    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:57:27.816601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816517    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-kube-api-access-fs4j6\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-kube-api-access-fs4j6\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816555    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-envoy\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816601 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816584    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-credential-socket\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816609    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-podinfo\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816659    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istiod-ca-cert\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816713    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-token\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-token\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816772    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-socket\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816809    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-certs\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.816917 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816838    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-data\") pod \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\" (UID: \"e0f7f72d-4152-4235-a2e2-ab6404fbcf33\") "
Oct 30 22:57:27.817269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.816907    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-credential-socket" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:27.817269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.817138    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-certs" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:27.817269 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.817255    5503 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.817453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.817275    5503 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-credential-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.817453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.817249    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-socket" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:27.817453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.817267    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istiod-ca-cert" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:27.817453 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.817357    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-data" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:27.818923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.818891    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-podinfo" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:57:27.819049 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.818905    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-kube-api-access-fs4j6" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "kube-api-access-fs4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:27.819156 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.819139    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-envoy" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:27.819268 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.819246    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-token" pod "e0f7f72d-4152-4235-a2e2-ab6404fbcf33" (UID: "e0f7f72d-4152-4235-a2e2-ab6404fbcf33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:27.912223 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.912188    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"]
Oct 30 22:57:27.918558 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.918523    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4j6\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-kube-api-access-fs4j6\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.918755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.918570    5503 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-envoy\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.918755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.918588    5503 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-podinfo\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.918755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.918603    5503 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istiod-ca-cert\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.918755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.918612    5503 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-token\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.918755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.918623    5503 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-workload-socket\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.918755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.918654    5503 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f7f72d-4152-4235-a2e2-ab6404fbcf33-istio-data\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:27.931677 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:27.931625    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"]
Oct 30 22:57:28.035575 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.035525    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:28.098307 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:57:28.098223    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:57:28.470630 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.470594    5503 generic.go:356] "Generic (PLEG): container finished" podID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerID="8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d" exitCode=0
Oct 30 22:57:28.470832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.470672    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"
Oct 30 22:57:28.470832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.470674    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" event={"ID":"e0f7f72d-4152-4235-a2e2-ab6404fbcf33","Type":"ContainerDied","Data":"8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d"}
Oct 30 22:57:28.470832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.470730    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k" event={"ID":"e0f7f72d-4152-4235-a2e2-ab6404fbcf33","Type":"ContainerDied","Data":"e2ad74f779f65ff16154a22e4b39c9818b5d8576651ea2d9b0bfa1fbb20b89f9"}
Oct 30 22:57:28.470832 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.470755    5503 scope.go:122] "RemoveContainer" containerID="8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d"
Oct 30 22:57:28.481151 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.481130    5503 scope.go:122] "RemoveContainer" containerID="465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894"
Oct 30 22:57:28.489494 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.489472    5503 scope.go:122] "RemoveContainer" containerID="8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d"
Oct 30 22:57:28.489767 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:57:28.489746    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d\": container with ID starting with 8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d not found: ID does not exist" containerID="8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d"
Oct 30 22:57:28.489830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.489776    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d"} err="failed to get container status \"8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d\": rpc error: code = NotFound desc = could not find container \"8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d\": container with ID starting with 8b8e99ce6776cd5055521420dd7da47e05b1f86761a02f89a33ce43207b1fb4d not found: ID does not exist"
Oct 30 22:57:28.489830 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.489792    5503 scope.go:122] "RemoveContainer" containerID="465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894"
Oct 30 22:57:28.490032 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:57:28.490014    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894\": container with ID starting with 465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894 not found: ID does not exist" containerID="465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894"
Oct 30 22:57:28.490087 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.490043    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894"} err="failed to get container status \"465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894\": rpc error: code = NotFound desc = could not find container \"465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894\": container with ID starting with 465e8c578b93b75d9d3421cc647509ce6c14d1b8a77f6f67c60687bfa3e2a894 not found: ID does not exist"
Oct 30 22:57:28.626963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.626930    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"]
Oct 30 22:57:28.673805 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:28.673767    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-74bb81-openshift-default-79ff795889-n5p4k"]
Oct 30 22:57:29.730238 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:29.730200    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:29.787973 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:29.787932    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" path="/var/lib/kubelet/pods/e0f7f72d-4152-4235-a2e2-ab6404fbcf33/volumes"
Oct 30 22:57:33.735912 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:33.735870    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:34.224086 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:34.224053    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:34.930481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:34.930445    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:34.971976 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:34.971940    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"]
Oct 30 22:57:35.397615 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:35.397523    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:35.597958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:35.597919    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:38.345835 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.345798    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"]
Oct 30 22:57:38.346251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346153    5503 cpu_manager.go:414] "containerMap: removing container" podUID="25f9d380-01af-437b-ad7f-dd90f4f9e079" containerName="revoke-keys"
Oct 30 22:57:38.346251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346163    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25f9d380-01af-437b-ad7f-dd90f4f9e079" containerName="revoke-keys"
Oct 30 22:57:38.346251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346174    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy"
Oct 30 22:57:38.346251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346179    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy"
Oct 30 22:57:38.346251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346186    5503 cpu_manager.go:414] "containerMap: removing container" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy"
Oct 30 22:57:38.346251 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346192    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy"
Oct 30 22:57:38.346452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346261    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="25f9d380-01af-437b-ad7f-dd90f4f9e079" containerName="revoke-keys"
Oct 30 22:57:38.346452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346269    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy"
Oct 30 22:57:38.346452 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.346275    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f7f72d-4152-4235-a2e2-ab6404fbcf33" containerName="istio-proxy"
Oct 30 22:57:38.349242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.349226    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.399034 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.399002    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:57:38.445541 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.445509    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:57:38.447353 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.447326    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"]
Oct 30 22:57:38.456572 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.456554    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:57:38.456855 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.456823    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wmbnv\""
Oct 30 22:57:38.507332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.507297    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.507332 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.507334    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhl82\" (UniqueName: \"kubernetes.io/projected/7ad0c358-1d98-4517-a2ad-d76870fc65e3-kube-api-access-lhl82\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.507540 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.507354    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.608320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.608223    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.608320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.608267    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhl82\" (UniqueName: \"kubernetes.io/projected/7ad0c358-1d98-4517-a2ad-d76870fc65e3-kube-api-access-lhl82\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.608320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.608291    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.608946 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.608923    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.610705 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.610683    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.667304 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.667266    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhl82\" (UniqueName: \"kubernetes.io/projected/7ad0c358-1d98-4517-a2ad-d76870fc65e3-kube-api-access-lhl82\") pod \"maas-api-revoke-sub-e360da17-nl46l\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:38.962360 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:38.962313    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:39.326564 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:39.326539    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"]
Oct 30 22:57:39.328867 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:57:39.328835    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad0c358_1d98_4517_a2ad_d76870fc65e3.slice/crio-120d90af1a285007fd0a35a52e0aaa229d621ab4be644d6dcd12fc0efd4a4a8a WatchSource:0}: Error finding container 120d90af1a285007fd0a35a52e0aaa229d621ab4be644d6dcd12fc0efd4a4a8a: Status 404 returned error can't find the container with id 120d90af1a285007fd0a35a52e0aaa229d621ab4be644d6dcd12fc0efd4a4a8a
Oct 30 22:57:39.531711 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:39.531674    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l" event={"ID":"7ad0c358-1d98-4517-a2ad-d76870fc65e3","Type":"ContainerStarted","Data":"120d90af1a285007fd0a35a52e0aaa229d621ab4be644d6dcd12fc0efd4a4a8a"}
Oct 30 22:57:39.973051 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:39.972991    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" podUID="545a76bc-1f02-4365-a785-fe0062df7c20" containerName="payload-processing" containerID="cri-o://15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031" gracePeriod=25
Oct 30 22:57:40.539197 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:40.539165    5503 generic.go:356] "Generic (PLEG): container finished" podID="7ad0c358-1d98-4517-a2ad-d76870fc65e3" containerID="fb40a8b11bf5e0bc5fada52dde16849721db30260a8c0e0904b9dbe2d54201fd" exitCode=0
Oct 30 22:57:40.539609 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:40.539247    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l" event={"ID":"7ad0c358-1d98-4517-a2ad-d76870fc65e3","Type":"ContainerDied","Data":"fb40a8b11bf5e0bc5fada52dde16849721db30260a8c0e0904b9dbe2d54201fd"}
Oct 30 22:57:41.668773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.668739    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:41.837995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.837922    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-xks-ca\") pod \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") "
Oct 30 22:57:41.838169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.837996    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-openshift-ca\") pod \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") "
Oct 30 22:57:41.838169 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.838080    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad0c358-1d98-4517-a2ad-d76870fc65e3-kube-api-access-lhl82\" (UniqueName: \"kubernetes.io/projected/7ad0c358-1d98-4517-a2ad-d76870fc65e3-kube-api-access-lhl82\") pod \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\" (UID: \"7ad0c358-1d98-4517-a2ad-d76870fc65e3\") "
Oct 30 22:57:41.838369 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.838349    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-openshift-ca" pod "7ad0c358-1d98-4517-a2ad-d76870fc65e3" (UID: "7ad0c358-1d98-4517-a2ad-d76870fc65e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:41.840028 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.839998    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-xks-ca" pod "7ad0c358-1d98-4517-a2ad-d76870fc65e3" (UID: "7ad0c358-1d98-4517-a2ad-d76870fc65e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:41.840148 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.840069    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad0c358-1d98-4517-a2ad-d76870fc65e3-kube-api-access-lhl82" pod "7ad0c358-1d98-4517-a2ad-d76870fc65e3" (UID: "7ad0c358-1d98-4517-a2ad-d76870fc65e3"). InnerVolumeSpecName "kube-api-access-lhl82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:41.938851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.938809    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhl82\" (UniqueName: \"kubernetes.io/projected/7ad0c358-1d98-4517-a2ad-d76870fc65e3-kube-api-access-lhl82\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.938851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.938843    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-xks-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:41.938851 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:41.938856    5503 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad0c358-1d98-4517-a2ad-d76870fc65e3-maas-api-openshift-ca\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:42.552320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:42.552283    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l"
Oct 30 22:57:42.552320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:42.552299    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360da17-nl46l" event={"ID":"7ad0c358-1d98-4517-a2ad-d76870fc65e3","Type":"ContainerDied","Data":"120d90af1a285007fd0a35a52e0aaa229d621ab4be644d6dcd12fc0efd4a4a8a"}
Oct 30 22:57:42.552554 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:42.552349    5503 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="120d90af1a285007fd0a35a52e0aaa229d621ab4be644d6dcd12fc0efd4a4a8a"
Oct 30 22:57:42.932429 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:42.932389    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main" containerID="cri-o://853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48" gracePeriod=45
Oct 30 22:57:42.950897 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:42.950850    5503 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main" containerID="cri-o://d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1" gracePeriod=45
Oct 30 22:57:43.206941 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.206912    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:57:43.209884 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.209862    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:57:43.353123 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353085    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-tmp-dir\") pod \"b406a09b-7afd-40cb-b013-25ca20c573df\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") "
Oct 30 22:57:43.353279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353140    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tls-certs\") pod \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") "
Oct 30 22:57:43.353279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353189    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b406a09b-7afd-40cb-b013-25ca20c573df-tls-certs\" (UniqueName: \"kubernetes.io/secret/b406a09b-7afd-40cb-b013-25ca20c573df-tls-certs\") pod \"b406a09b-7afd-40cb-b013-25ca20c573df\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") "
Oct 30 22:57:43.353279 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353236    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-dshm\") pod \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") "
Oct 30 22:57:43.353493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353319    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-model-cache\") pod \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") "
Oct 30 22:57:43.353493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353359    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-home\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-home\") pod \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") "
Oct 30 22:57:43.353493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353393    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tmp-dir\") pod \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") "
Oct 30 22:57:43.353493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353421    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-tmp-dir" pod "b406a09b-7afd-40cb-b013-25ca20c573df" (UID: "b406a09b-7afd-40cb-b013-25ca20c573df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.353493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353431    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-home\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-home\") pod \"b406a09b-7afd-40cb-b013-25ca20c573df\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") "
Oct 30 22:57:43.353493 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353485    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-kube-api-access-j878k\" (UniqueName: \"kubernetes.io/projected/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-kube-api-access-j878k\") pod \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\" (UID: \"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe\") "
Oct 30 22:57:43.353848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353548    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b406a09b-7afd-40cb-b013-25ca20c573df-kube-api-access-5s8pc\" (UniqueName: \"kubernetes.io/projected/b406a09b-7afd-40cb-b013-25ca20c573df-kube-api-access-5s8pc\") pod \"b406a09b-7afd-40cb-b013-25ca20c573df\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") "
Oct 30 22:57:43.353848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353623    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-dshm\") pod \"b406a09b-7afd-40cb-b013-25ca20c573df\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") "
Oct 30 22:57:43.353848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353658    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-home" pod "b406a09b-7afd-40cb-b013-25ca20c573df" (UID: "b406a09b-7afd-40cb-b013-25ca20c573df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.353848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353678    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-model-cache\") pod \"b406a09b-7afd-40cb-b013-25ca20c573df\" (UID: \"b406a09b-7afd-40cb-b013-25ca20c573df\") "
Oct 30 22:57:43.353848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353695    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-model-cache" pod "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" (UID: "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.353848 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353820    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-home" pod "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" (UID: "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.354198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.353900    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tmp-dir" pod "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" (UID: "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.354198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.354036    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-model-cache" pod "b406a09b-7afd-40cb-b013-25ca20c573df" (UID: "b406a09b-7afd-40cb-b013-25ca20c573df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.354198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.354051    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.354198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.354073    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.354198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.354088    5503 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-home\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.354198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.354103    5503 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-tmp-dir\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.354198 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.354116    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.356227 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.356181    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tls-certs" pod "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" (UID: "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:43.356476 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.356448    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-dshm" pod "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" (UID: "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.356561 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.356484    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-dshm" pod "b406a09b-7afd-40cb-b013-25ca20c573df" (UID: "b406a09b-7afd-40cb-b013-25ca20c573df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:43.356751 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.356727    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-kube-api-access-j878k" pod "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" (UID: "11159a97-55c0-4e97-8bb5-1a0ae53ca1fe"). InnerVolumeSpecName "kube-api-access-j878k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:43.357177 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.357152    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b406a09b-7afd-40cb-b013-25ca20c573df-kube-api-access-5s8pc" pod "b406a09b-7afd-40cb-b013-25ca20c573df" (UID: "b406a09b-7afd-40cb-b013-25ca20c573df"). InnerVolumeSpecName "kube-api-access-5s8pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:43.357299 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.357274    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b406a09b-7afd-40cb-b013-25ca20c573df-tls-certs" pod "b406a09b-7afd-40cb-b013-25ca20c573df" (UID: "b406a09b-7afd-40cb-b013-25ca20c573df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:43.455119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.455038    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.455119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.455071    5503 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b406a09b-7afd-40cb-b013-25ca20c573df-tls-certs\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.455119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.455084    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.455119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.455097    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j878k\" (UniqueName: \"kubernetes.io/projected/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe-kube-api-access-j878k\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.455119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.455110    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8pc\" (UniqueName: \"kubernetes.io/projected/b406a09b-7afd-40cb-b013-25ca20c573df-kube-api-access-5s8pc\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.455119 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.455122    5503 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-dshm\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.455483 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.455134    5503 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b406a09b-7afd-40cb-b013-25ca20c573df-model-cache\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:57:43.558837 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.558802    5503 generic.go:356] "Generic (PLEG): container finished" podID="b406a09b-7afd-40cb-b013-25ca20c573df" containerID="853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48" exitCode=0
Oct 30 22:57:43.559033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.558864    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"
Oct 30 22:57:43.559033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.558893    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" event={"ID":"b406a09b-7afd-40cb-b013-25ca20c573df","Type":"ContainerDied","Data":"853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48"}
Oct 30 22:57:43.559033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.558938    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc" event={"ID":"b406a09b-7afd-40cb-b013-25ca20c573df","Type":"ContainerDied","Data":"64d1c3b81cccc9916507a7b687fd2574bc05a8881f9b35a0c7fd92a813b5dc09"}
Oct 30 22:57:43.559033 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.558955    5503 scope.go:122] "RemoveContainer" containerID="853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48"
Oct 30 22:57:43.560396 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.560373    5503 generic.go:356] "Generic (PLEG): container finished" podID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerID="d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1" exitCode=0
Oct 30 22:57:43.560505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.560416    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"
Oct 30 22:57:43.560581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.560419    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" event={"ID":"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe","Type":"ContainerDied","Data":"d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1"}
Oct 30 22:57:43.560581 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.560570    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf" event={"ID":"11159a97-55c0-4e97-8bb5-1a0ae53ca1fe","Type":"ContainerDied","Data":"773f26201ff39d00498e9b5e796c7b8b3a91152c188fe063b097f9075e11d72b"}
Oct 30 22:57:43.569834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.569815    5503 scope.go:122] "RemoveContainer" containerID="853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48"
Oct 30 22:57:43.570190 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:57:43.570166    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48\": container with ID starting with 853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48 not found: ID does not exist" containerID="853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48"
Oct 30 22:57:43.570298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.570197    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48"} err="failed to get container status \"853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48\": rpc error: code = NotFound desc = could not find container \"853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48\": container with ID starting with 853254dec88f5e813a72ee38fcf95c16da48a37dbb5610780b87a4386836cd48 not found: ID does not exist"
Oct 30 22:57:43.570298 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.570216    5503 scope.go:122] "RemoveContainer" containerID="d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1"
Oct 30 22:57:43.578755 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.578738    5503 scope.go:122] "RemoveContainer" containerID="d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1"
Oct 30 22:57:43.578996 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:57:43.578981    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1\": container with ID starting with d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1 not found: ID does not exist" containerID="d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1"
Oct 30 22:57:43.579048 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.579003    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1"} err="failed to get container status \"d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1\": rpc error: code = NotFound desc = could not find container \"d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1\": container with ID starting with d283b37d95bb0478a7568b49b4caad164940c3b3eb1d5de5f95d432dd8b4d4b1 not found: ID does not exist"
Oct 30 22:57:43.714725 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.714617    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"]
Oct 30 22:57:43.718587 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.718562    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/simulator-w0-d09fff-kserve-7bfdff55d-crtgf"]
Oct 30 22:57:43.789084 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.789049    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" path="/var/lib/kubelet/pods/11159a97-55c0-4e97-8bb5-1a0ae53ca1fe/volumes"
Oct 30 22:57:43.822250 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.822220    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"]
Oct 30 22:57:43.824813 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.824791    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d09fff/distinct-w0-d09fff-kserve-b89f5cc66-d2fnc"]
Oct 30 22:57:43.987940 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:43.987857    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" podUID="545a76bc-1f02-4365-a785-fe0062df7c20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:45.787542 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:45.787506    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" path="/var/lib/kubelet/pods/b406a09b-7afd-40cb-b013-25ca20c573df/volumes"
Oct 30 22:57:53.726684 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:53.726630    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:53.987328 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:53.987240    5503 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" podUID="545a76bc-1f02-4365-a785-fe0062df7c20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:55.171456 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:55.171420    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:55.185762 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:55.185732    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:57:55.445276 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:55.445198    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:56.128900 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:57:56.128866    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:57:58.098369 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:57:58.098331    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:58:00.215265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.215241    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 22:58:00.281495 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.281439    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/545a76bc-1f02-4365-a785-fe0062df7c20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545a76bc-1f02-4365-a785-fe0062df7c20-plugins-config-volume\") pod \"545a76bc-1f02-4365-a785-fe0062df7c20\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") "
Oct 30 22:58:00.281621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.281499    5503 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/545a76bc-1f02-4365-a785-fe0062df7c20-kube-api-access-g8jd4\" (UniqueName: \"kubernetes.io/projected/545a76bc-1f02-4365-a785-fe0062df7c20-kube-api-access-g8jd4\") pod \"545a76bc-1f02-4365-a785-fe0062df7c20\" (UID: \"545a76bc-1f02-4365-a785-fe0062df7c20\") "
Oct 30 22:58:00.281858 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.281839    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545a76bc-1f02-4365-a785-fe0062df7c20-plugins-config-volume" pod "545a76bc-1f02-4365-a785-fe0062df7c20" (UID: "545a76bc-1f02-4365-a785-fe0062df7c20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:58:00.283427 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.283407    5503 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545a76bc-1f02-4365-a785-fe0062df7c20-kube-api-access-g8jd4" pod "545a76bc-1f02-4365-a785-fe0062df7c20" (UID: "545a76bc-1f02-4365-a785-fe0062df7c20"). InnerVolumeSpecName "kube-api-access-g8jd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:00.382152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.382116    5503 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/545a76bc-1f02-4365-a785-fe0062df7c20-plugins-config-volume\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:00.382152 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.382145    5503 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jd4\" (UniqueName: \"kubernetes.io/projected/545a76bc-1f02-4365-a785-fe0062df7c20-kube-api-access-g8jd4\") on node \"ip-10-0-141-86.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:58:00.654621 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.654587    5503 generic.go:356] "Generic (PLEG): container finished" podID="545a76bc-1f02-4365-a785-fe0062df7c20" containerID="15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031" exitCode=0
Oct 30 22:58:00.654834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.654650    5503 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"
Oct 30 22:58:00.654834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.654668    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" event={"ID":"545a76bc-1f02-4365-a785-fe0062df7c20","Type":"ContainerDied","Data":"15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031"}
Oct 30 22:58:00.654834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.654722    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw" event={"ID":"545a76bc-1f02-4365-a785-fe0062df7c20","Type":"ContainerDied","Data":"fc917d3cedb25668367a268bed00b9407bf75ac3c3d8f43c08bf59b484c82694"}
Oct 30 22:58:00.654834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.654746    5503 scope.go:122] "RemoveContainer" containerID="15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031"
Oct 30 22:58:00.664404 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.664383    5503 scope.go:122] "RemoveContainer" containerID="15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031"
Oct 30 22:58:00.664700 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:58:00.664678    5503 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031\": container with ID starting with 15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031 not found: ID does not exist" containerID="15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031"
Oct 30 22:58:00.664766 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.664712    5503 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031"} err="failed to get container status \"15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031\": rpc error: code = NotFound desc = could not find container \"15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031\": container with ID starting with 15c912980f67997c33b4298cbbf4f83174859f9d946e8fb9d14913864f938031 not found: ID does not exist"
Oct 30 22:58:00.810482 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.810441    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"]
Oct 30 22:58:00.815024 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:00.814996    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d09fff-7bfdd689b5-kjmmw"]
Oct 30 22:58:01.786934 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:01.786901    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="545a76bc-1f02-4365-a785-fe0062df7c20" path="/var/lib/kubelet/pods/545a76bc-1f02-4365-a785-fe0062df7c20/volumes"
Oct 30 22:58:02.495796 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:02.495750    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:02.656277 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:02.656245    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:02.876276 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:02.876191    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:05.011486 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:05.011460    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jj7d9_d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520/kuadrant-console-plugin/0.log"
Oct 30 22:58:05.589576 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:05.589541    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:05.795706 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:05.795621    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:05.996898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:05.996865    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:06.418265 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:06.418234    5503 scope.go:122] "RemoveContainer" containerID="9a30d7ba69df4946c7b6947974f18d25d590238ba773c4191f1b4566a30de6fc"
Oct 30 22:58:06.426859 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:06.426840    5503 scope.go:122] "RemoveContainer" containerID="854dc601df678fa31ebbf716a910da237f162a4a6b9192ecaef2a96f59898142"
Oct 30 22:58:06.434489 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:06.434468    5503 scope.go:122] "RemoveContainer" containerID="a5d79c7e61420ac2abe4e9b86e73683125233ae491ec40842bc84103118117ea"
Oct 30 22:58:06.441863 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:06.441847    5503 scope.go:122] "RemoveContainer" containerID="00eadb3b057e6665b1cef88a1e8c2a932eae337d71b01c64a5c1afa898ae6614"
Oct 30 22:58:06.449505 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:06.449488    5503 scope.go:122] "RemoveContainer" containerID="bd701a5248886b641e5f7047296484889ff372c07e72ac84d78f070b8932ef6b"
Oct 30 22:58:06.460471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:06.460455    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-x544p_a7a9e65b-ce49-42f4-93fe-79cf3bc2769e/limitador/0.log"
Oct 30 22:58:08.519944 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:08.519916    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:58:09.132624 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:09.132587    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-x544p"]
Oct 30 22:58:09.582053 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:09.582019    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:58:09.852358 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:09.852290    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:58:11.734962 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:11.734929    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:22.047362 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:22.047333    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:58:23.196833 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:23.196801    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:58:23.466407 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:23.466333    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:58:24.550153 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:24.550126    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ef731d1-w9ngv_477f6424-6017-4c48-ab49-37d8111f5ba2/revoke-keys/0.log"
Oct 30 22:58:24.777905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:24.777878    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-59f77dfe-wb2hx_ebf907df-42f2-4c00-ba1a-60405bdfc3c7/revoke-keys/0.log"
Oct 30 22:58:25.002140 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:25.002109    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85818335-qrgwd_60f7b283-dc0a-49d3-ac55-8d316cc9d93f/revoke-keys/0.log"
Oct 30 22:58:25.233778 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:25.233748    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85e70155-c4lk8_999c1eff-d672-4255-92f5-eb3076411333/revoke-keys/0.log"
Oct 30 22:58:25.680888 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:25.680854    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9a3d0123-zsh72_a5e43d4f-b1f5-4502-bae8-827e24270207/revoke-keys/0.log"
Oct 30 22:58:25.906560 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:25.906527    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d90c6b12-kgwcs_e59b1fa2-41ab-459f-b139-3cfc31c7d521/revoke-keys/0.log"
Oct 30 22:58:26.141383 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:26.141284    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e360da17-nl46l_7ad0c358-1d98-4517-a2ad-d76870fc65e3/revoke-keys/0.log"
Oct 30 22:58:28.098160 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:58:28.098126    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:58:28.638553 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:28.638527    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jj7d9_d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520/kuadrant-console-plugin/0.log"
Oct 30 22:58:29.764834 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:29.764802    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-x544p_a7a9e65b-ce49-42f4-93fe-79cf3bc2769e/limitador/0.log"
Oct 30 22:58:31.179320 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:31.179291    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wjg9j_ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26/discovery/0.log"
Oct 30 22:58:31.638023 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:31.637946    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d969c5987-wm52k_373ba953-29a3-4811-aea7-9dd8dfa3b086/kube-auth-proxy/0.log"
Oct 30 22:58:31.863143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:31.863115    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ltvq9_bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe/istio-proxy/0.log"
Oct 30 22:58:34.033792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:34.033757    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:35.258852 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:35.258820    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:45.527066 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:45.527027    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:45.811815 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:45.811722    5503 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log" limit=1024
Oct 30 22:58:51.095471 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:51.095427    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"]
Oct 30 22:58:51.096161 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:51.096135    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3d0123-zsh72"]
Oct 30 22:58:51.788481 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:51.788444    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e43d4f-b1f5-4502-bae8-827e24270207" path="/var/lib/kubelet/pods/a5e43d4f-b1f5-4502-bae8-827e24270207/volumes"
Oct 30 22:58:52.045760 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:52.045658    5503 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"]
Oct 30 22:58:52.091827 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:52.091793    5503 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f77dfe-wb2hx"]
Oct 30 22:58:53.787878 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:53.787848    5503 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebf907df-42f2-4c00-ba1a-60405bdfc3c7" path="/var/lib/kubelet/pods/ebf907df-42f2-4c00-ba1a-60405bdfc3c7/volumes"
Oct 30 22:58:56.514387 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:56.514358    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lwfgr_14e12922-2bce-41b4-935d-5d3708d1694e/global-pull-secret-syncer/0.log"
Oct 30 22:58:57.933155 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:57.933117    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:58:58.097248 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:58:58.097212    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"
Oct 30 22:58:58.474462 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:58.474412    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-blztt_cb4b1c1f-cf24-41bc-a3b9-50ada9e9eb67/konnectivity-agent/0.log"
Oct 30 22:58:59.056204 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:59.056173    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:58:59.701895 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:59.701854    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:58:59.822261 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:58:59.822233    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:59:00.594219 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:00.594181    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-86.us-west-2.compute.internal_25a4aaee5f241beed648c981baba69bb/haproxy/0.log"
Oct 30 22:59:00.622732 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:00.622704    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:59:00.736905 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:00.736878    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:59:03.557650 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:03.557606    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:59:05.189991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:05.189959    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:59:05.478537 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:05.478452    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:59:05.526921 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:05.526895    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:59:06.497242 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:06.497206    5503 scope.go:122] "RemoveContainer" containerID="c3492554465f6c607cb7c8055c8ff14d26d3981a35ab5f5dde0a7c3e45fc8bed"
Oct 30 22:59:06.508273 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:06.508227    5503 scope.go:122] "RemoveContainer" containerID="fbfdfb693ebd4e5f0be32c1cec5fb05cd729b597b8719980a623d1c30eec56df"
Oct 30 22:59:06.518191 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:06.518170    5503 scope.go:122] "RemoveContainer" containerID="a6e4aae3bc14cc4c432d566ce849430a210c7c86c51e2ebf4ae9c6c8001ac046"
Oct 30 22:59:06.526709 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:06.526688    5503 scope.go:122] "RemoveContainer" containerID="e175993ee61deb6363b39be7225d35f53b513fc671e1e015beda791666b27b94"
Oct 30 22:59:06.860782 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:06.860691    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jj7d9_d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520/kuadrant-console-plugin/0.log"
Oct 30 22:59:07.118996 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.118915    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:59:07.277594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.277561    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:59:07.458384 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458345    5503 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"]
Oct 30 22:59:07.458738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458724    5503 cpu_manager.go:414] "containerMap: removing container" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main"
Oct 30 22:59:07.458792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458739    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main"
Oct 30 22:59:07.458792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458758    5503 cpu_manager.go:414] "containerMap: removing container" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main"
Oct 30 22:59:07.458792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458763    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main"
Oct 30 22:59:07.458792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458782    5503 cpu_manager.go:414] "containerMap: removing container" podUID="545a76bc-1f02-4365-a785-fe0062df7c20" containerName="payload-processing"
Oct 30 22:59:07.458792 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458788    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="545a76bc-1f02-4365-a785-fe0062df7c20" containerName="payload-processing"
Oct 30 22:59:07.458963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458796    5503 cpu_manager.go:414] "containerMap: removing container" podUID="7ad0c358-1d98-4517-a2ad-d76870fc65e3" containerName="revoke-keys"
Oct 30 22:59:07.458963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458800    5503 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad0c358-1d98-4517-a2ad-d76870fc65e3" containerName="revoke-keys"
Oct 30 22:59:07.458963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458856    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="b406a09b-7afd-40cb-b013-25ca20c573df" containerName="main"
Oct 30 22:59:07.458963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458864    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="11159a97-55c0-4e97-8bb5-1a0ae53ca1fe" containerName="main"
Oct 30 22:59:07.458963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458874    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="545a76bc-1f02-4365-a785-fe0062df7c20" containerName="payload-processing"
Oct 30 22:59:07.458963 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.458880    5503 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad0c358-1d98-4517-a2ad-d76870fc65e3" containerName="revoke-keys"
Oct 30 22:59:07.463310 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.463290    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.552081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.552045    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-proc\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.552081 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.552081    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-sys\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.552589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.552104    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-lib-modules\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.552589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.552261    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rbh\" (UniqueName: \"kubernetes.io/projected/d4c7a742-8581-49a9-9ab8-eca59df5c97e-kube-api-access-72rbh\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.552589 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.552357    5503 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-podres\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.559923 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.559898    5503 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7d8m9\"/\"default-dockercfg-qb7pg\""
Oct 30 22:59:07.560064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.559971    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7d8m9\"/\"openshift-service-ca.crt\""
Oct 30 22:59:07.560064 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.560016    5503 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7d8m9\"/\"kube-root-ca.crt\""
Oct 30 22:59:07.561398 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.561379    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"]
Oct 30 22:59:07.653005 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.652973    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-lib-modules\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653018    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rbh\" (UniqueName: \"kubernetes.io/projected/d4c7a742-8581-49a9-9ab8-eca59df5c97e-kube-api-access-72rbh\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653062    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-podres\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653112    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-proc\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653124    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-lib-modules\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653136    5503 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-sys\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653183    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-proc\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653199 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653194    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-sys\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.653466 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.653194    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4c7a742-8581-49a9-9ab8-eca59df5c97e-podres\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.711991 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.711921    5503 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rbh\" (UniqueName: \"kubernetes.io/projected/d4c7a742-8581-49a9-9ab8-eca59df5c97e-kube-api-access-72rbh\") pod \"perf-node-gather-daemonset-r4shh\" (UID: \"d4c7a742-8581-49a9-9ab8-eca59df5c97e\") " pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:07.774995 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:07.774961    5503 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:08.143019 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:08.142990    5503 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"]
Oct 30 22:59:08.145849 ip-10-0-141-86 kubenswrapper[5503]: W1030 22:59:08.145815    5503 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4c7a742_8581_49a9_9ab8_eca59df5c97e.slice/crio-5572db1ce7c494004c8e0f93902370659a6e15be92fca4e312b4a7669fdfbf04 WatchSource:0}: Error finding container 5572db1ce7c494004c8e0f93902370659a6e15be92fca4e312b4a7669fdfbf04: Status 404 returned error can't find the container with id 5572db1ce7c494004c8e0f93902370659a6e15be92fca4e312b4a7669fdfbf04
Oct 30 22:59:08.253491 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:08.253460    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-x544p_a7a9e65b-ce49-42f4-93fe-79cf3bc2769e/limitador/0.log"
Oct 30 22:59:09.018989 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:09.018940    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh" event={"ID":"d4c7a742-8581-49a9-9ab8-eca59df5c97e","Type":"ContainerStarted","Data":"0b74b9c25c9da03e968df205b5c867c3eabc88bef87f3d4e7d7c8a29d6a58dd3"}
Oct 30 22:59:09.019415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:09.019005    5503 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh" event={"ID":"d4c7a742-8581-49a9-9ab8-eca59df5c97e","Type":"ContainerStarted","Data":"5572db1ce7c494004c8e0f93902370659a6e15be92fca4e312b4a7669fdfbf04"}
Oct 30 22:59:09.019415 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:09.019032    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:09.131393 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:09.131335    5503 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh" podStartSLOduration=2.131316554 podStartE2EDuration="2.131316554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:09.129483272 +0000 UTC m=+4325.464060052" watchObservedRunningTime="2026-10-30 22:59:09.131316554 +0000 UTC m=+4325.465893323"
Oct 30 22:59:09.348094 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:09.348006    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:59:10.802547 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:10.802510    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:59:10.967810 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:10.967776    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:59:11.208958 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:11.208926    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-88frl_8d32001d-5d67-45a9-b4f9-1571c4b9d093/manager/0.log"
Oct 30 22:59:11.264272 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:11.264240    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6747dd9f-877b-41f4-b482-b2d4500a6cff/alertmanager/0.log"
Oct 30 22:59:11.673692 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:11.673664    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6747dd9f-877b-41f4-b482-b2d4500a6cff/config-reloader/0.log"
Oct 30 22:59:12.088898 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:12.088799    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6747dd9f-877b-41f4-b482-b2d4500a6cff/kube-rbac-proxy-web/0.log"
Oct 30 22:59:12.646738 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:12.646700    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6747dd9f-877b-41f4-b482-b2d4500a6cff/kube-rbac-proxy/0.log"
Oct 30 22:59:12.784185 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:12.784151    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-p67bl_5fb64a84-3645-47f9-9fb7-f898b1951963/manager/0.log"
Oct 30 22:59:12.926473 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:12.926442    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wc4d4_44057d31-f693-45ab-99e5-c04ec9b02101/proxy/0.log"
Oct 30 22:59:13.055435 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:13.055407    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6747dd9f-877b-41f4-b482-b2d4500a6cff/kube-rbac-proxy-metric/0.log"
Oct 30 22:59:13.464478 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:13.464448    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6747dd9f-877b-41f4-b482-b2d4500a6cff/prom-label-proxy/0.log"
Oct 30 22:59:13.872309 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:13.872218    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6747dd9f-877b-41f4-b482-b2d4500a6cff/init-config-reloader/0.log"
Oct 30 22:59:14.703773 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:14.703743    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-p7fkb_d87bf1f4-0cf2-4c1d-a513-2c6862b68970/kube-state-metrics/0.log"
Oct 30 22:59:15.034143 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:15.034047    5503 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7d8m9/perf-node-gather-daemonset-r4shh"
Oct 30 22:59:15.113672 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:15.113624    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-p7fkb_d87bf1f4-0cf2-4c1d-a513-2c6862b68970/kube-rbac-proxy-main/0.log"
Oct 30 22:59:15.410886 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:15.410856    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wjg9j_ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26/discovery/0.log"
Oct 30 22:59:15.521860 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:15.521832    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-p7fkb_d87bf1f4-0cf2-4c1d-a513-2c6862b68970/kube-rbac-proxy-self/0.log"
Oct 30 22:59:15.722110 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:15.722020    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d969c5987-wm52k_373ba953-29a3-4811-aea7-9dd8dfa3b086/kube-auth-proxy/0.log"
Oct 30 22:59:15.931883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:15.931851    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ltvq9_bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe/istio-proxy/0.log"
Oct 30 22:59:16.329839 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:16.329811    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-2vq2h_8748e87c-9dcc-442b-b739-820276e15f03/monitoring-plugin/0.log"
Oct 30 22:59:17.250883 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:17.250847    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wjg9j_ac1fe945-0f1b-49dd-ab82-e43fe1ec6d26/discovery/0.log"
Oct 30 22:59:17.568990 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:17.568906    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d969c5987-wm52k_373ba953-29a3-4811-aea7-9dd8dfa3b086/kube-auth-proxy/0.log"
Oct 30 22:59:17.786449 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:17.786414    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6f54779779-ltvq9_bb1bca9b-e5fe-4448-bed4-e7bfd3005bbe/istio-proxy/0.log"
Oct 30 22:59:17.982225 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:17.982186    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8z59d_c388501b-fe35-4373-ba59-7391b3c79fb5/node-exporter/0.log"
Oct 30 22:59:18.399607 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:18.399511    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8z59d_c388501b-fe35-4373-ba59-7391b3c79fb5/kube-rbac-proxy/0.log"
Oct 30 22:59:18.808262 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:18.808233    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8z59d_c388501b-fe35-4373-ba59-7391b3c79fb5/init-textfile/0.log"
Oct 30 22:59:22.865770 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:22.865744    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jj7d9_d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520/kuadrant-console-plugin/0.log"
Oct 30 22:59:23.824338 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:23.824309    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-x544p_a7a9e65b-ce49-42f4-93fe-79cf3bc2769e/limitador/0.log"
Oct 30 22:59:24.479296 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:24.479267    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qcvc2_91a3fbfd-209a-4c30-8fb9-9e4812d02f27/prometheus-operator/0.log"
Oct 30 22:59:24.886877 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:24.886805    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qcvc2_91a3fbfd-209a-4c30-8fb9-9e4812d02f27/kube-rbac-proxy/0.log"
Oct 30 22:59:25.494658 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:25.494612    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jj7d9_d2c57c9f-1e2a-4f6c-bb53-8dcbfeae6520/kuadrant-console-plugin/0.log"
Oct 30 22:59:25.700097 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:25.700069    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7458fbb99c-q7l58_0b8454ef-998f-4574-ae4e-ab0c094a7562/telemeter-client/0.log"
Oct 30 22:59:26.115126 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:26.115084    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7458fbb99c-q7l58_0b8454ef-998f-4574-ae4e-ab0c094a7562/reload/0.log"
Oct 30 22:59:26.484594 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:26.484568    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-x544p_a7a9e65b-ce49-42f4-93fe-79cf3bc2769e/limitador/0.log"
Oct 30 22:59:26.528044 ip-10-0-141-86 kubenswrapper[5503]: I1030 22:59:26.528018    5503 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7458fbb99c-q7l58_0b8454ef-998f-4574-ae4e-ab0c094a7562/kube-rbac-proxy/0.log"
Oct 30 22:59:28.097367 ip-10-0-141-86 kubenswrapper[5503]: E1030 22:59:28.097328    5503 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2sdgq" containerName="extractor"